Imports Overhaul and Operations Center

The Problem

Operations is a very important area in passare, configurations, setups, imports, scheduling is mostly done by them, unfortunately, they have been using the same tools for years which were made with limitations, due to the size passare had at the moment those were implemented.

For example, Data Imports, these different imports were implemented years ago and they were prepared to handle a few hundred records per import and from time to time.

Nowadays, passare has a constant flow of data importing for clients and because we have grown exponentially, more and more data is needed for new and existing customers, changing the requirement from a few hundred records from time to time, to several thousand records every day.

With the limitations we currently have in our import process, this becomes literally, and unbearable task:

• Cannot import more than 2 thousand records at the same time

• Can't upload several files at once

• It is synchronous: even though processing is in the background, there is no way to know the progress of the import once you move out of the import page, this is locking users to stay on the page while the import is happening

• Many times, an import fail due to uncontrolled stacking: Because imports is becoming the most annoying task in Passare, users try to save some time by uploading several imports in different tabs, this generates parallel processing which increases the chances of the import failing, which means more and more wasted time

• There are several pre-requisites depending on the import, and until one of the rows fail due to missing one of these, you won't know exactly what you are missing, this mean users can stay up to 2 hours waiting for an import to finish, to just then fail, so they have to start over

• There is no history of what was imported and when

• Users cannot change their organizations while importing as that can make imports to import into the wrong organization

• A long list of etc.

Out Proposal

I would like to treat operations with a new and clear section for them Called "Operations" Or "Operations Center", this section will include all existing super admin sections, including the new scheduled jobs section, but most importantly, a NEW imports page.

New Operations Page

• This page should look like the Financials page, where there is a left side menu

• This page will be accessible to users that are super admin, as a link at the end of the current "Admin" header menu, called "Operations"

• Left side menu should include all existing super admin menu items

• An additional "Data Imports" menu item should be added alphabetically ordered

Data Imports Page

• Users need to be able to choose what type of import they want to do including:

  1. Cases

     1. Basic Information

     2. Veteran Information

     3. Family And Friends

     4. Events

     5. Contract Information

     6. Payments

  2. Rolodex

  3. Price List

  4. Client Service Types

  5. Checks

  6. Bank Transactions

• Users need to be able to choose the organization they want to import to, without having to switch their current organization

• Users need to be able to upload several files as a single import request, rather than uploading one by one

• Users need to be able to get off the imports page, and be able to get back to the page and see the progress of their import request

• Users should be able to stop an import request

• Users should be able to see a history of import requests

  1. Users should be able to see files to import/imported

  2. Users should be able to see the result of the import per file

     1. Included sidekiq tyle error for development to troublehoot

  3. User should be able to see who requested the import, for which organization and when

• Users should be able to clearly see if there are any missing pre-requisites to import

  1. This will depend on the import they are trying to do, and it should prevent from submitting the import request if there is missing data or missing pre-requisites

• Users should be able to setup different criteria to be defined for example:

  1. Fail entire import on first fail

  2. Import as much as possible (keep importing the rest of the file even if a record fails to import)

  3. Be able to analyze the import data in order to find potential problems before the import is triggered (this will depend on the import type itself, but will be most likely validations for missing data or duplicated data)

• Passare backend should be ablet o manage how imports work to avoid:

  1. Parallel processing

  2. Slowness and queue locking

  3. Unnecessary failures

Benefits of this idea

1. Reduce drastically the time needed to do imports both for Operations, Client Services and Premium Services and avoid wasted time, saving money and improving our operations people quality of life and their work.

2. Better troubleshooting for development when an import is happening, leading to faster bug fixing related to imports.

3. Be able to prevent problems by stopping current, and future data imports if necessary