Featured Post

Sava Gromov on the scene

You may wonder who the heck is the unknown Mr. Gromov?  To put it bluntly, he is the main character of Sovereign, the new emerging spa...

Sunday, May 19, 2019

Django to the rescue

Recently, I saw thread on reddit, in which the replacement of Excel by Python was discussed. I skimmed through and realized (as expected) that everybody consider just one aspect of Excel, the data analysis.


To my huge disappointment,  nobody ever mentioned the second terrible aspect of Excel, the data storage.






Before you get started explaining me the differences between Excel and Access, let me assure you I am aware of them. Sadly, I am surrounded by coworkers who not only do not know what is Access, but who would be probably surprised if being told that they are abusing Excel in the worst manner possible.
About ten years ago, I adopted MS Access for managing data of some of our processes. Please make no mistake, this is not story about how Access greatly improved our data management because it did not. Actually, I grew tired by it over the time for three reasons:

  • availability,
  • bugs,
  • and scalability.
At the beginning, I was enjoying the easiness of creating model and designing forms. But as soon as I made my coworkers to use my database file shared on network drive, we run into several problems.  With growing number of production records, the main table became sluggish and not responding. 
Lately, the crushes occurred so often I didn't know if I would be able to finish my work.  
The last drop in my chalice was decision of our IT department not to have Access license for new computers. 

From my point of view, I got the opportunity and excuse  to make my data-driven application anew. While my coworkers were drowning in the unmaintainable hell of excel files, littered over the local network  in all possible paths, I start coding.
My target was to create thin client, hosted on linux server in our local network, available to all who have a browser installed on their computers - translated to plain English,  the app should be (and is) available to everyone, not only the lucky ones with licensed Access.   
As this post is intended for Pythonistas, I am omitting further explanation of what Python, Django, and relational database are. I intent to write another post for non-technical audience later. Nevertheless, the situation in chemical processing facility might be beyond comprehension of average software engineer, who never had the opportunity to witness real world in first person.
My dear friends, it is an awful unmanageable mess. No defined structures, several wrongly implemented systems, a lot of word and excel files, duplicated, incorrect, and inconsistent data everywhere, insufficient backups, no security and no training whatsoever. Simply put, something far far away from the environment you are used to. 


How did I proceed?

To make everything as simple as possible I developed an application based mainly on Django admin as  replacement of Access. Mind you I am no professional coder, so I my time was rather limited. To avoid complicated settings I just use sqlite database as there was no indication it should not be enough for my purposes. Furthermore, the backup of single file proved easier than backup of  full blown database like PostgreSQL or MySQL. 
To make it short, my approach worked. Soon enough I made a prototype application that had functionality to replace my old database and many of our excel files. Our management liked that (well, they have no better alternative, anyway) and my coworkers...
Do you expect me to lie about how thrilled they were? They stay absolutely indifferent and nobody annoys me asking me to improve their lives with automating boring things - or teach them the importancy of math.

To avoid any pessimistic undertone, let me assure you I was able to get rid of many incorrect and redundant data using Python and Django. The most import thing to note is that data in those files were result of long-term abuse rather than usage of MS Excel. 

No comments:

Post a Comment