Operations is a very important area in passare, configurations, setups, imports, scheduling is mostly done by them, unfortunately, they have been using the same tools for years which were made with limitations, due to the size passare had at the moment those were implemented.
For example, Data Imports, these different imports were implemented years ago and they were prepared to handle a few hundred records per import and from time to time.
Nowadays, passare has a constant flow of data importing for clients and because we have grown exponentially, more and more data is needed for new and existing customers, changing the requirement from a few hundred records from time to time, to several thousand records every day.
With the limitations we currently have in our import process, this becomes literally, and unbearable task:
Cannot import more than 2 thousand records at the same time
Can't upload several files at once
It is synchronous: even though processing is in the background, there is no way to know the progress of the import once you move out of the import page, this is locking users to stay on the page while the import is happening
Many times, an import fail due to uncontrolled stacking: Because imports is becoming the most annoying task in Passare, users try to save some time by uploading several imports in different tabs, this generates parallel processing which increases the chances of the import failing, which means more and more wasted time
There are several pre-requisites depending on the import, and until one of the rows fail due to missing one of these, you won't know exactly what you are missing, this mean users can stay up to 2 hours waiting for an import to finish, to just then fail, so they have to start over
There is no history of what was imported and when
Users cannot change their organizations while importing as that can make imports to import into the wrong organization
A long list of etc.
I would like to treat operations with a new and clear section for them Called "Operations" Or "Operations Center", this section will include all existing super admin sections, including the new scheduled jobs section, but most importantly, a NEW imports page.
This page should look like the Financials page, where there is a left side menu
This page will be accessible to users that are super admin, as a link at the end of the current "Admin" header menu, called "Operations"
Left side menu should include all existing super admin menu items
An additional "Data Imports" menu item should be added alphabetically ordered
Users need to be able to choose what type of import they want to do including:
Cases
Basic Information
Veteran Information
Family And Friends
Events
Contract Information
Payments
Rolodex
Price List
Client Service Types
Checks
Bank Transactions
Users need to be able to choose the organization they want to import to, without having to switch their current organization
Users need to be able to upload several files as a single import request, rather than uploading one by one
Users need to be able to get off the imports page, and be able to get back to the page and see the progress of their import request
Users should be able to stop an import request
Users should be able to see a history of import requests
Users should be able to see files to import/imported
Users should be able to see the result of the import per file
Included sidekiq tyle error for development to troublehoot
User should be able to see who requested the import, for which organization and when
Users should be able to clearly see if there are any missing pre-requisites to import
This will depend on the import they are trying to do, and it should prevent from submitting the import request if there is missing data or missing pre-requisites
Users should be able to setup different criteria to be defined for example:
Fail entire import on first fail
Import as much as possible (keep importing the rest of the file even if a record fails to import)
Be able to analyze the import data in order to find potential problems before the import is triggered (this will depend on the import type itself, but will be most likely validations for missing data or duplicated data)
Passare backend should be ablet o manage how imports work to avoid:
Parallel processing
Slowness and queue locking
Unnecessary failures
Reduce drastically the time needed to do imports both for Operations, Client Services and Premium Services and avoid wasted time, saving money and improving our operations people quality of life and their work.
Better troubleshooting for development when an import is happening, leading to faster bug fixing related to imports.
Be able to prevent problems by stopping current, and future data imports if necessary
Your name | Matias Laino |
Hack Week Project Owner | Matias Laino |