Autoload Information
Interfolio supports autoloading for various data types, including base data (e.g., grants, scholarly activity data, and custom sections) as well as non-base data (e.g., profiles and activities). All autoloads are dependent on having Secure File Transfer Protocol (SFTP) access to the Interfolio servers to check for new files uploaded to the SFTP server and load them into Faculty Activity Reporting (FAR). This can be done prior to or in conjunction with the request for autoload setup. Once configured by Interfolio Support, base data autoloads are configured to automatically process new files uploaded to the SFTP server on a scheduled basis via SFTP. Data types not supported by autoload at this time are processed manually upon request or according to a pre-arranged schedule with the Technical Services team.
Note that once a data file is approved for autoloading, the file structure (file names, column names, column order, and formatting) will need to stay consistent. Only non-base data can have a name change. If needed, please send a support request before making the change.
SFTP is not required for the initial bulk upload of data, but is required for autoloaded data types. While email submissions are acceptable, using SFTP is the recommended best practice for all data types to ensure secure handling.
If you do not have SFTP access, email Interfolio-support@elsevier.com with your request including the name of the section to be autoloaded, a sample of the data file(s), the email addresses that need to receive the autoload notification, the autoload frequency required, and the day/time for the process to run (e.g. every Friday at 5 am EST).
Autoload Information
Supported Autoloads & Naming Conventions
Naming Convention
All upload files should follow the naming format Faculty180_<upload_type>_<yyyy>-<mm>-<dd>_<sequence>.csv.
For example, the file name Faculty180_CoursesTaught_2015-08-11_01.csv indicates a Courses Taught file uploaded on August 11, 2015, and it is the first file of this type received on that date. If a second Courses Taught was to be sent on the same day, the sequence would change to Faculty180_CoursesTaught_2015-08-11_02.csv
| Available Upload Types for Base Data | ||
|---|---|---|
| Supported Import | <upload_type> | Example Filename |
| Faculty | faculty | Faculty180_faculty_2019-08-11_01.csv |
| Faculty Classifications | faculty classifications | Faculty180_facultyclassifications_2019-08-11_01.csv |
| Current Positions | current positions | Faculty180_currentpositions_2019-08-11_01.csv |
| Course Prefixes | prefix | Faculty180_prefix_2019-08-11_01.csv |
| Courses | courses | Faculty180_courses_2019-08-11_01.csv |
| Courses Taught | courses taught | Faculty180_coursestaught_2019-08-11_01.csv |
| Available Upload Types for Non-Base Data | ||
|---|---|---|
| Supported Import | <upload_type> | Example Filename |
| Scholarship | abstract, book, digitalmedia, journalarticle, etc | Faculty180_abstract_2019-08-11_01.csv |
| Grants | Award | Faculty180_award_2019-08-11_01.csv |
| Grants | Award (if providing collaborator file) | Faculty180_award_collaborator_2019-08-11_01.csv |
| Grants | Proposal | Faculty180_proposal_2019-08-11_01.csv |
| Grants | Proposal (if providing collaborator file) | Faculty180_proposal_collaborator_2019-08-11_01.csv |
| Custom Section | Most file types within FAR | Faculty180_[section_name]_2019-08-11_01.csv |
Coming Soon
- Degrees
- Other Institutional Service
- Institutional Committees
- Work Experience
- Attachments
Time of Import
Data files can be uploaded to the SFTP server at any time. Most institutions load files once a week or once a month, but files can be scheduled for autoload on a weekly, monthly, or quarterly basis. If a file is not available at the scheduled time, the process will simply skip that run and attempt again during the next scheduled interval.
| Data Type | Import Time | Other Details |
|---|---|---|
| Base Data | Imports are scheduled for processing at 2:30 AM EST/3:30 EDT daily daily. | If no data is found, or if an Error Message appears, then the import will not occur. |
| Scholarship & Grants | By default, the system checks for files on the 1st, 10th, 20th, and 30th of the month. | Import schedules can be customized upon request. |
| Custom Sections | By default, the system checks for files on the 5th, 15th, and 25th of the month. |
Custom Sections are most sections in FAR. Import schedules can be customized upon request. When editing an input form, each section will display either 'Reorder' for Default sections or 'Custom Section Reorder' for custom sections. |
File Location
All files will be located in a directory created by Interfolio Support. Each SFTP directory has a default folder structure with each of your databases configured below (e.g. /autoload/interfolio_university/).
- Base Data: /autoload/[database_name]/[data_type]
- Non-Base Data: /incoming/[database_name]/[data_type]
- e.g. Grants files should be uploaded to: /incoming/[database_name]/grants
Supported file types are .csv (comma-delimited) and .txt (tab-delimited) only. File types that are not supported and should not be included are .doc, .docx, .pdf, .xls, .xlsx, subdirectories, etc.
Do not direct files to the first level directory folders (e.g. /incoming, /processed, /autoload, /outgoing), but rather to the appropriate subdirectory folder for the database.
| First-Level Directory Folder | Details | Subdirectory Folders |
|---|---|---|
| incoming | All non-base data files, such as grants, advising, professional development, and more, that are not supported by the base-data autoload. |
Files from the client (e.g. grants, advising, scholarship) will be directed to /incoming/[database_name] if the file is to be loaded into production.
Data types (e.g. scholarly, grants, degrees, custom) will follow the path /incoming/[database_name]/[data_type]. |
| processed | Base data autoload files for faculty, current position, faculty classification, prefix, courses, and courses taught that are supported by the base-data autoload. | Files are moved to /processed/[database_name] after processing. |
| autoload | Autoload has to be set up in order to support this processing. Please reach out to Interfolio-support@elsevier.com if your autoloads need to be set up or reconfigured. |
/autoload/[database_name] is the production database directory for base data autoload (only files for faculty, current position, faculty classification, prefix, courses, courses taught). NotificationsAfter autoload file processing, notifications are sent to the email addresses that have been added to /autoload/[databasename]/notification.txt.
Users can edit the addresses directly at any time by downloading the file, making the necessary edits, and re-uploading. Individual addresses should be followed by a line break. Please do not edit/delete the @interfolio.com address listed. |
|
outgoing For Interfolio use only. |
Any non-base data files, such as file logs and Interfolio-generated file extracts (limited scope), that are not supported by the base data autoload. |
/outgoing/[databasename]/logs contains non-base-data processed file logs (recent version), such as grants, advising, scholarship, etc.
/outgoing/[databasename]/old_logs contains non-base data processed file logs (older version). |
After processing non-base data files into production, those files will be placed under the old_files subdirectory.
Base Data Upload Configuration
To set up autoloads review file processing settings:
- Navigate to File Configuration to configure the Upload Form.
- Navigate to File Settings to conduct Bulk Data Uploads.
Non-base data files (except for grants) are not supported for autoloading at this time. Base-data autoload settings require an Interfolio System Administrator to save the settings. Please submit a support request for assistance in setting up the base-data autoload feature. If changes to the saved settings are needed, please submit a support request for assistance.
Base Data Type Autoload Order
- Units
- Faculty
- Faculty Classifications
- Course Prefixes
- Courses
- Courses Taught
Autoload Logs & Notifications
When an autoload process runs, it generates two log files stored on the SFTP server in the /outgoing/logs directory. The success log records all records that were processed successfully, indicating whether each was inserted as new, updated, or left unchanged. The error log captures records that failed to load, providing a reason for each failure and additional details if available.
Files will be processed if they are in the required file format (.csv or .txt), follow the specified column order (as defined on the setup in the database), and have been successfully uploaded via SFTP to the autoload folder for the given database.
Upon completion of the autoload, an email notification can be sent to designated recipients. Emails in this file could include FAR administration or the entity that creates files for automatic upload. This email, sent from noreply@f180-prod.localdomain, will include the name of the files processed, the name of the section loaded, the total number of new records, updated records, records not changed and errors. This setup ensures that relevant stakeholders stay informed about the status of each autoload operation.
UTF-8 encoded file formats are supported. If particular uploaded .CSV file(s) are not encoded in UTF-8, re-save the files as UTF-8 in order for the data to be read correctly by the data base. It is recommended to provide a delta file, instead of a file containing the full data set. A delta file will include new records and records that have changed since the last load. This will help to reduce processing time and improve the accuracy of the updates.
For base data, each directory under /autoload/ contains a notification.txt file, which lists email addresses (one per line) of users to be notified once the process completes. These emails will contain information on the upload, including counts of new, duplicate, and failed records as well as a list of lines that were duplicate or failed. Emails in this file could include FAR administration or the entity that creates files for automatic upload.
Once granted SFTP access, users can edit recipient notifications by navigating to autoloads > notifications.txt, downloading and editing the file, then re-uploading it. Please do not remove the @interfolio.com address included in the notification.txt file.
For non-base data, notification recipients are managed by Interfolio’s Technical Services team; to update the recipient list, contact Interfolio-support@elsevier.com.
Deleting Activity & Profile Data in Bulk
As part of your institution’s process for loading data in bulk to Interfolio, you may wish to:
- Bulk delete previously bulk-imported data (e.g. remove grants data (proposals) after they have been awarded, correct erroneous data uploaded in bulk, etc).
- Delete all bulk-loaded data except for a known set of records.
These bulk deletion capabilities are now available through the Interfolio Data Machine (IDM). Supported data types include custom sections, grants, and scholarships. In order for records to be deleted in this workflow, the following requirements must be met:
- They must have been previously bulk loaded via Technical Services or the IDM.
- The file must follow the naming & format conventions outlined below.
Deleting unlocked records carries a risk of data loss. The ideal candidates for bulk deletion are sections where:
- Your institution is loading bulk data with the lock flag enabled.
- Edits are restricted via the user interface (UI).
Interested in implementing an automated bulk deletion into your institution’s workflow? Reach out to our Technical Services team (Data Integration Services) via a ticket to interfolio-support@elsevier.com. Provide the name of the FAR section for which you’d like to delete data (if known).
File Templates & Naming Conventions
- Records to Delete: faculty180_[name_of_section]_DELETE_yyyy-mm-dd.csv
All records in this file will be deleted.
- Records to Keep: faculty180_[name of section]_RETAIN_yyyy-mm-dd.csv
All records not specified in this file will be deleted. Only the included records will be retained.
File Format
All files must be UTF-8-encoded CSVs
- Grants section:
| Column 1 | Column 2 |
| <Award Record ID> | <Faculty ID> |
| Column 1 | Column 2 |
| <Proposal Record ID> | <Faculty ID> |
- Scholary & Custom sections:
| Column 1 | Column 2 |
| <Record ID> | <Faculty ID> |
SFTP Location
Place the ‘Delete’ or ‘Retain’ CSV in the same SFTP location as the corresponding bulk data file (for example, /incoming/grants).
Timing
Once the file is processed, logs will be available under /outgoing/logs. Interfolio can configure any recipient to receive a notification email upon completion.
- The ‘Delete’ file will be processed before the primary file, when both are present.
By default, both files (the primary and deletion files) will need to be present in order for the Delete file to process. For example, if you wish to delete data from a custom section, the data file must be present alongside the Delete file in order for the latter job to execute. This can be customized on a case-by-case basis.
- The ‘Retain’ file will be processed before the primary file, when both are present.
By default, both files (the primary and renention files) will need to be present in order for the Retain file to process. For example, if you wish to delete data from a custom section, the data file must be present alongside the Retain file in order for the latter job to execute. This can be customized on a case-by-case basis.
- If more than one ‘Delete’ file is present, IDM will process only the most recent, based on the date in the file name.
- If more than one ‘Retain’ file is present, IDM will process only the most recent, based on the date in the file name.
- Our system will check for the file at the same time as your regular cadence for the data type in question.
Check out our Data Load Checklist article for further details on what to include when preparing a file to be loaded via the Technical Services team.
FAQs
What happens when a file is processed?
For Supported Base Data Autoload Types that Meet Requirements
- After processing, files are moved from the autoload folder to the processed folder.
- Unprocessed files (those that do not meet requirements) remain in the autoload folder. The system will continue to try to process the bad file.
- Please review the notifications that are sent. These failed files will be noted and the file should be deleted from the directory folder.
For Supported Non-Base-Data Autoload Types that Meet Requirements
- After processing, files are moved to a folder within the same directory called /old_files.
- Unprocessed files (those that do not meet requirements) remain in the /incoming folder.
- Please review the notifications that are sent and the outgoing logs for more details. These failed files will be noted and the file(s) should be removed from the directory folder or modified to meet requirements.
What happens when file modifications are needed?
For Supported Base Data Autoload Types
- It is possible to start with a file that only has the required data field and then add additional fields as needed.
- This will require a change to the setup. Please submit a support request before making changes to the current file.
For Supported Non-Base Data Autoload Types
- Contact the Technical Services team via Interfolio-support@elsevier.com
What is the threshold for the “delete” or “retain” file and can it be customized?
By default, our system will not process the records of a “delete” or “retain” file if the number of impacted records exceeds 20% of the total data previously loaded in bulk to the section in question.
This threshold can be adjusted up or down according to your institution's needs.
What happens if a “delete” or “retain” file is shared inadvertently, or contains incorrect information?
Your institution is responsible for sending accurate information to Interfolio regarding what records to keep and which to delete. In the event that records are impacted inadvertently, please contact Support immediately. The data can be restored via a bulk load provided by your institution (when applicable), or via a full-database restore to the latest available backup. Please note that the latter is a complete database restore, so all data in your database will be restored to an earlier state. Manual edits to records bulk imported to FAR will be lost.
What file encoding should I use for uploads, and should I upload full data sets or delta files?
All files should be uploaded via SFTP to the designated autoload folder associated with your database. For efficient processing and to maintain data accuracy, it is recommended to upload delta files, which include only new records and updates, rather than full data sets. Using delta files reduces processing time and ensures that only the necessary information is updated in the system.
- Acceptable file formats are .csv or .txt, and the column order must align with the specifications provided during your database setup
- Files must be encoded in UTF-8 to ensure the system reads the data correctly. If your .csv files are not in UTF-8, you should re-save or convert them before uploading.