#! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! #! FME_PYTHON_VERSION 35 DEFAULT_MACRO domain GUI OPTIONAL TEXT domain : DEFAULT_MACRO username GUI OPTIONAL TEXT username : DEFAULT_MACRO password GUI OPTIONAL PASSWORD password : DEFAULT_MACRO dataset_uid ykrz-u6jt GUI OPTIONAL TEXT dataset_uid : DEFAULT_MACRO upload_config GUI OPTIONAL TEXT upload_config : DEFAULT_MACRO csv_name GUI OPTIONAL FILENAME_EXISTING csv_name |*.csv : DEFAULT_MACRO csv_folder $(FME_MF_DIR_USERTYPED) GUI OPTIONAL DIRNAME csv_folder : #! START_HEADER #! START_WB_HEADER READER_TYPE MULTI_READER WRITER_TYPE NULL WRITER_KEYWORD NULL_DEST NULL_DEST_DATASET null #! END_WB_HEADER #! START_WB_HEADER #! END_WB_HEADER #! END_HEADER LOG_FILENAME "$(FME_MF_DIR)Socrata_PythonAutomateThis_v1.0.log" LOG_APPEND NO LOG_TIMINGS YES LOG_FILTER_MASK -1 LOG_MAX_FEATURES 200 LOG_MAX_RECORDED_FEATURES 200 FME_REPROJECTION_ENGINE FME FME_IMPLICIT_CSMAP_REPROJECTION_MODE Auto FME_GEOMETRY_HANDLING Enhanced FME_STROKE_MAX_DEVIATION 0 FME_PASSWORD_PARAMETER_LIST password # ------------------------------------------------------------------------- MULTI_READER_CONTINUE_ON_READER_FAILURE No # ------------------------------------------------------------------------- MACRO WORKSPACE_NAME Socrata_PythonAutomateThis_v1.0 MACRO FME_VIEWER_APP fmedatainspector DEFAULT_MACRO WB_CURRENT_CONTEXT # ------------------------------------------------------------------------- Tcl2 proc Creator_CoordSysRemover {} { global FME_CoordSys; set FME_CoordSys {}; } MACRO Creator_XML NOT_ACTIVATED MACRO Creator_CLASSIC NOT_ACTIVATED MACRO Creator_2D3D 2D_GEOMETRY MACRO Creator_COORDS INCLUDE [ if { {Geometry Object} == {Geometry Object} } { puts {MACRO Creator_XML *} } ] INCLUDE [ if { {Geometry Object} == {2D Coordinate List} } { puts {MACRO Creator_2D3D 2D_GEOMETRY}; puts {MACRO Creator_CLASSIC *} } ] INCLUDE [ if { {Geometry Object} == {3D Coordinate List} } { puts {MACRO Creator_2D3D 3D_GEOMETRY}; puts {MACRO Creator_CLASSIC *} } ] INCLUDE [ if { {Geometry Object} == {2D Min/Max Box} } { set comment { We need to turn the COORDS which are minX minY maxX maxY into a full polygon list of coordinates }; set splitCoords [split [string trim {}]]; if { [llength $splitCoords] > 4} { set trimmedCoords {}; foreach item $splitCoords { if { $item != {} } {lappend trimmedCoords $item} }; set splitCoords $trimmedCoords; }; if { [llength $splitCoords] != 4 } { error {Creator: Coordinate list is expected to be a space delimited list of four numbers as 'minx miny maxx maxy' - `' is invalid}; }; set minX [lindex $splitCoords 0]; set minY [lindex $splitCoords 1]; set maxX [lindex $splitCoords 2]; set maxY [lindex $splitCoords 3]; puts "MACRO Creator_COORDS $minX $minY $minX $maxY $maxX $maxY $maxX $minY $minX $minY"; puts {MACRO Creator_2D3D 2D_GEOMETRY}; puts {MACRO Creator_CLASSIC *} } ] FACTORY_DEF $(Creator_XML) CreationFactory FACTORY_NAME Creator_XML_Creator CREATE_AT_END no OUTPUT FEATURE_TYPE _____CREATED______ @Geometry(FROM_ENCODED_STRING,?xmlversion=1.0encoding=US_ASCIIstandalone=no?geometrydimension=2nullgeometry) FACTORY_DEF $(Creator_CLASSIC) CreationFactory FACTORY_NAME Creator_CLASSIC_Creator $(Creator_2D3D) $(Creator_COORDS) CREATE_AT_END no OUTPUT FEATURE_TYPE _____CREATED______ FACTORY_DEF * TeeFactory FACTORY_NAME Creator_Cloner INPUT FEATURE_TYPE _____CREATED______ @Tcl2(Creator_CoordSysRemover) @CoordSys() NUMBER_OF_COPIES 1 COPY_NUMBER_ATTRIBUTE "_creation_instance" OUTPUT FEATURE_TYPE Creator_CREATED fme_feature_type Creator FACTORY_DEF * BranchingFactory FACTORY_NAME "Creator_CREATED Brancher -1 8" INPUT FEATURE_TYPE Creator_CREATED TARGET_FACTORY "$(WB_CURRENT_CONTEXT)_CREATOR_BRANCH_TARGET" MAXIMUM_COUNT None OUTPUT PASSED FEATURE_TYPE * # ------------------------------------------------------------------------- FACTORY_DEF * TeeFactory FACTORY_NAME "$(WB_CURRENT_CONTEXT)_CREATOR_BRANCH_TARGET" INPUT FEATURE_TYPE * OUTPUT FEATURE_TYPE * # ------------------------------------------------------------------------- MACRO FeatureReader_OUTPUT_PORTS_ENCODED CSV MACRO FeatureReader_DIRECTIVES ENFORCE_STRICT_SCHEMA,No,SCAN_ADDITIONAL_FIELDS,No,SCAN_MAX_ROWS,1000,FIELD_NAMES_LINE,1,PRESERVE_SOURCE_ENCODING,No,QUERY_FEATURE_TYPES_FOR_MERGE_FILTERS,Yes,SKIP_DUPLICATE_DELIMITERS,No,USE_CUSTOM_SCHEMA,NO,FIELD_QUALIFIER_CHARACTER,quote,CASE_SENSITIVE_FIELD_NAMES,Yes,READ_SKIPPED_LINES,No,SCAN_TYPES,No,TRIM_FIELD_NAME_QUALIFIERS,Yes,FIELD_QUALIFIER_ESCAPE_CHARACTER,quote,TRIM_FIELD_VALUE_QUALIFIERS,Yes,DELIMITER,auto,FEATURE_TYPE_NAME,FORMAT_NAME,DATA_START_LINE,2 # Always provide an INTERACTION, otherwise the factory defaults to ENVELOPE_INTERSECTS INCLUDE [if { ( {NONE} == {} ) || ( {($INTERACT_OPTIONS)} == {} ) } { puts {MACRO FCTQUERY_INTERACTION_LINE FCTQUERY_INTERACTION NONE}; } else { puts {MACRO FCTQUERY_INTERACTION_LINE FCTQUERY_INTERACTION "NONE"}; } ] # Consolidate the attribute merge options to what the factory expects DEFAULT_MACRO FeatureReader_COMBINE_ATTRS INCLUDE [ if { {RESULT_ONLY} == {MERGE} } { puts "MACRO FeatureReader_COMBINE_ATTRS "; } else { puts "MACRO FeatureReader_COMBINE_ATTRS RESULT_ONLY"; }; ] INCLUDE [ puts {DEFAULT_MACRO FeatureReaderDataset_FeatureReader @EvaluateExpression(FDIV,STRING_ENCODED,$(FME_MF_DIR_ENCODED)Enchantments_Camp_Sites_2018.csv,FeatureReader)}; ] FACTORY_DEF * QueryFactory FACTORY_NAME FeatureReader INPUT FEATURE_TYPE Creator_CREATED $(FCTQUERY_INTERACTION_LINE) COMBINE_ATTRIBUTES $(FeatureReader_COMBINE_ATTRS) QUERYFCT_ATTRIBUTE_PREFIX COMBINE_GEOMETRY RESULT_ONLY ENABLE_CACHE NO QUERYFCT_TABLE_SEPARATOR SPACE READER_TYPE CSV2 READER_DATASET "$(FeatureReaderDataset_FeatureReader)" QUERYFCT_IDS "" READER_DIRECTIVES META_MACROS,SourceDATASET_GROUPSourceFEATURE_TYPE_NAMEFORMAT_NAMESourceFIELDS_GROUPSourceDELIMITERautoSourceFIELD_NAMES_LINE1SourceDATA_START_LINE2SourceADVANCED_GROUPSourceFIELD_SEPARATION_GROUPSourceSKIP_DUPLICATE_DELIMITERSNoSourceFIELD_QUALIFIER_CHARACTERquoteSourceFIELD_QUALIFIER_ESCAPE_CHARACTERquoteSourceFIELD_NAMES_GROUPSourceTRIM_FIELD_NAME_QUALIFIERSYesSourceCASE_SENSITIVE_FIELD_NAMESYesSourceENFORCE_STRICT_SCHEMANoSourceFIELD_VALUES_GROUPSourceTRIM_FIELD_VALUE_QUALIFIERSYesSourceENCODING_GROUPSourceENCODINGSourcePRESERVE_SOURCE_ENCODINGNoSourceSKIPPED_LINES_GROUPSourceREAD_SKIPPED_LINESNoSourceSCHEMA_GENERATION_GROUPSourceSCAN_MAX_ROWS1000SourceSCAN_TYPESNoSourceSCAN_ADDITIONAL_FIELDSNoSourcePREVIEW_PLACEHOLDERSourceUSE_CUSTOM_SCHEMANOSourceSCHEMASourceEXPOSE_ATTRS_GROUPSourceCSV2_EXPOSE_FORMAT_ATTRSSourceQUERY_FEATURE_TYPES_FOR_MERGE_FILTERSYes,METAFILE,CSV2 QUERYFCT_OUTPUT "BASED_ON_CONNECTIONS" CONTINUE_ON_READER_ERROR YES QUERYFCT_RESULT_TAGS $(FeatureReader_OUTPUT_PORTS_ENCODED) QUERYFCT_SET_FME_FEATURE_TYPE YES READER_PARAMS_WWJD $(FeatureReader_DIRECTIVES) TREAT_READER_PARAM_AMPERSANDS_AS_LITERALS YES OUTPUT READER_ERROR FEATURE_TYPE FeatureReader_ OUTPUT CSV FEATURE_TYPE FeatureReader_CSV DEFAULT_MACRO _WB_BYPASS_TERMINATION No FACTORY_DEF * TeeFactory FACTORY_NAME FeatureReader_ INPUT FEATURE_TYPE FeatureReader_ OUTPUT FAILED FEATURE_TYPE * @Abort(ENCODED, FeatureReaderoutputaRejectedfeature.TocontinuetranslationwhenfeaturesarerejectedchangeWorkspaceParametersTranslationRejectedFeatureHandlingtoContinueTranslation) # ------------------------------------------------------------------------- INCLUDE [ puts {DEFAULT_MACRO FeatureWriterDataset_FeatureWriter @EvaluateExpression(FDIV,STRING_ENCODED,$(csv_folder$encode),FeatureWriter)}; ] FACTORY_DEF * WriterFactory FACTORY_NAME FeatureWriter WRITER_TYPE CSV2 WRITER_DATASET "$(FeatureWriterDataset_FeatureWriter)" WRITER_SETTINGS RUNTIME_MACROS,DATASET_GROUPEXTENSIONcsvEXISTING_FILE_HANDLINGYesFIELDS_GROUPDELIMITERcommaWRITE_FIELD_NAMESfirstrowENCODING_GROUPENCODINGWRITE_BOM_FOR_UNICODEYesADVANCED_GROUPQUALIFY_FIELD_NAMESif_neededQUALIFY_FIELD_VALUESif_neededFIELD_QUALIFIER_CHARACTERquoteFIELD_QUALIFIER_ESCAPE_CHARACTERquoteROW_TERMINATIONSystemWRITE_SCHEMA_FILENOEXISTING_SCHEMA_FILE_HANDLINGUnusedCOORDINATE_SYSTEM_GRANULARITYFEATURE_TYPE,METAFILE,CSV2 WRITER_METAFILE ATTRIBUTE_CASE,ANY,ATTRIBUTE_INVALID_CHARS,*,ATTRIBUTE_LENGTH,255,ATTR_TYPE_MAP,stringfme_bufferstringfme_binarybufferstringfme_xmlstringfme_jsonstringfme_varcharwidthstringfme_varbinarywidthstringfme_charwidthstringfme_binarywidthstringfme_datetimestringfme_timestringfme_datestringfme_real64real64fme_real64x_coordinatefme_real64y_coordinatefme_real64z_coordinatefme_real64stringfme_real32real32fme_real32stringfme_int64int64fme_int64stringfme_int32int32fme_int32stringfme_int16int16fme_int16stringfme_int8int8fme_int8stringfme_uint64uint64fme_uint64stringfme_uint32uint32fme_uint32stringfme_uint16uint16fme_uint16stringfme_uint8uint8fme_uint8stringfme_booleanbooleanfme_booleanstringfme_decimalwidthdecimal,DEST_ILLEGAL_ATTR_LIST,,FEATURE_TYPE_CASE,ANY,FEATURE_TYPE_INVALID_CHARS,:?*|,FEATURE_TYPE_LENGTH,0,FEATURE_TYPE_LENGTH_INCLUDES_PREFIX,false,FEATURE_TYPE_RESERVED_WORDS,,FORMAT_METAFILE,$(FME_HOME_ENCODED)metafileCSV2.fmf,FORMAT_NAME,CSV2,GEOM_MAP,csv_feature_tablefme_feature_tablecsv_nonefme_no_geomcsv_pointfme_pointcsv_nonefme_linecsv_nonefme_polygoncsv_nonefme_textcsv_nonefme_ellipsecsv_nonefme_arccsv_nonefme_rectanglecsv_nonefme_rounded_rectanglefme_no_mapfme_no_geomfme_no_mapfme_no_mapcsv_nonefme_collectioncsv_nonefme_rastercsv_nonefme_surfacecsv_nonefme_solidcsv_nonefme_point_cloud,READER_ATTR_INDEX_TYPES,,SOURCE,no,SUPPORTS_FEAT_TYPE_FANOUT,yes,SUPPORTS_MULTI_GEOM,yes,WORKBENCH_CANNED_SCHEMA,,WRITER,CSV2,WRITER_ATTR_INDEX_TYPES,,WRITER_DEFLINE_PARMS,GUINAMEDGROUPcsv_dataset_groupcsv_existing_file_handlingDatasetGUICHOICEcsv_existing_file_handlingYes%NoOverwriteExistingFileYesGUINAMEDGROUPcsv_fields_groupcsv_write_field_namesFieldsGUILOOKUP_CHOICEcsv_write_field_namesIfspaceWritingspaceFirstspaceRowfirstrow%Yesyes%NonoWriteFieldNamesRowfirstrow,WRITER_DEF_LINE_TEMPLATE,FME_GEN_GROUP_NAMEcsv_existing_file_handlingYescsv_write_field_namesfirstrow,WRITER_FORMAT_PARAMETER,ADVANCED_PARMSCSV2_OUT_QUALIFY_FIELD_NAMESCSV2_OUT_QUALIFY_FIELD_VALUESCSV2_OUT_FIELD_QUALIFIER_CHARACTERCSV2_OUT_FIELD_QUALIFIER_ESCAPE_CHARACTERCSV2_OUT_ROW_TERMINATIONCSV2_OUT_WRITE_SCHEMA_FILECSV2_OUT_EXISTING_SCHEMA_FILE_HANDLINGDEFAULT_ATTR_TYPEstringDATASET_NAMEcsvfileFEATURE_TYPE_NAMECSVFileFEATURE_TYPE_DEFAULT_NAMEFile1READER_DATASET_HINTSelecttheCSVfilesWRITER_DATASET_HINTSpecifyafolderfortheCSVfileATTRIBUTE_READINGDEFLINE,WRITER_HAS_DEFLINE_ATTRS,yes WRITER_FEATURE_TYPES csv_upload:CSV,ftp_feature_type_name_exp,$(csv_name),ftp_writer,CSV2,ftp_dynamic_schema,no,ftp_dynamic_feature_type_name_type,DYN_SCHEMA_PROP_AUTO,ftp_dynamic_geometry_type,DYN_SCHEMA_PROP_AUTO,ftp_dynamic_schema_def_name_type,DYN_SCHEMA_PROP_AUTO,ftp_dynamic_schema_sources,ltUnusedgt,ftp_attribute_source,0,ftp_user_attributes,TRAILspaceNUMBERstringDISTURBEDspaceAREAstringXstringYstringthe_geomstringRANGERstringSURVEYspaceDATEstring,ftp_user_attribute_values,,ftp_format_parameters,csv_dataset_groupcsv_existing_file_handlingYescsv_fields_groupcsv_write_field_namesfirstrow WRITER_PARAMS DELIMITER,comma,WRITE_FIELD_NAMES,firstrow,COORDINATE_SYSTEM_GRANULARITY,FEATURE_TYPE,ROW_TERMINATION,System,WRITE_SCHEMA_FILE,NO,FIELD_QUALIFIER_CHARACTER,quote,EXTENSION,csv,QUALIFY_FIELD_NAMES,if_needed,FIELD_QUALIFIER_ESCAPE_CHARACTER,quote,QUALIFY_FIELD_VALUES,if_needed,WRITE_BOM_FOR_UNICODE,Yes,EXISTING_FILE_HANDLING,Yes DATASET_ATTR _dataset FEATURE_TYPE_LIST_ATTR _feature_type TOTAL_FEATURES_WRITTEN_ATTR _total_features_written OUTPUT_PORTS "" INPUT CSV FEATURE_TYPE FeatureReader_CSV @FeatureType(ENCODED,@EvaluateExpression(FDIV,STRING_ENCODED,$(csv_name$encode),FeatureWriter)) @SupplyAttributes(ENCODED,fme_template_feature_type,CSV) OUTPUT SUMMARY FEATURE_TYPE FeatureWriter_SUMMARY # ------------------------------------------------------------------------- FME_PYTHON_PATH "$(FME_MF_DIR)" FACTORY_DEF * PythonFactory FACTORY_NAME AutomateThisPython INPUT FEATURE_TYPE FeatureWriter_SUMMARY SYMBOL_NAME FeatureProcessor PYTHON_NAMESPACE FMEOBJECTS SOURCE_CODE importfmeimportfmeobjectsfromsocrata.authorizationimportAuthorizationfromsocrataimportSocrataimportsys#TemplateFunctioninterface:#Whenusingthisfunctionmakesureitsnameissetasthevalueof#theClassorFunctiontoProcessFeaturestransformerparameterdefprocessFeaturefeature:pass#TemplateClassInterface:#Whenusingthisclassmakesureitsnameissetasthevalueof#theClassorFunctiontoProcessFeaturestransformerparameterclassFeatureProcessorobject:def__init__self:passdefinputselffeature:auth=AuthorizationFME_MacroValuesdomainFME_MacroValuesusernameFME_MacroValuespasswordsocrata=Socrataauthokview=socrata.views.lookupFME_MacroValuesdataset_uidassertokview#slashbetweencsvfolderandcsvnamemaybeduplicativeornotneededdependingonoperatingsystemwithopenFME_MacroValuescsv_folder++FME_MacroValuescsv_name+.csvrbasmy_file:okjob=socrata.using_configFME_MacroValuesupload_configview.csvmy_fileassertokjob#Thesenext3linesareoptional-oncethejobisstartedfromthepreviouslinethe#scriptcanexitthesenextlinesjustblockuntilthejobcompletesassertokjobokjob=job.wait_for_finishprogress=lambdajob:printJobprogress:job.attributesstatus#sys.exit0ifjob.attributesstatus==successfulelse1self.pyoutputfeaturedefcloseself:pass OUTPUT PYOUTPUT FEATURE_TYPE AutomateThisPython_OUTPUT FACTORY_DEF * TeeFactory FACTORY_NAME "AutomateThisPython OUTPUT Transformer Output Nuker" INPUT FEATURE_TYPE AutomateThisPython_OUTPUT # ------------------------------------------------------------------------- FACTORY_DEF * RoutingFactory FACTORY_NAME "Destination Feature Type Routing Correlator" COMMAND_PARM_EVALUATION SINGLE_PASS INPUT FEATURE_TYPE * FEATURE_TYPE_ATTRIBUTE __wb_out_feat_type__ OUTPUT ROUTED FEATURE_TYPE * OUTPUT NOT_ROUTED FEATURE_TYPE __nuke_me__ @Tcl2("FME_StatMessage 818059 [FME_GetAttribute fme_template_feature_type] 818060 818061 fme_warn") # ------------------------------------------------------------------------- FACTORY_DEF * TeeFactory FACTORY_NAME "Final Output Nuker" INPUT FEATURE_TYPE __nuke_me__