Yes that isn't a thing; its a term I came up with, and I sincerely hope no one ever has to encounter such a situation.
So, someone wrote a complex data processing pipeline, that read from some tables in a DB, processed the data in small and large ways, finally producing a bunch of variables that were passed into a downstream ML model. No documentation was produced, and the developer had exited the company.
The problem: because of some high stakes issues with said pipeline's results, we needed to (quickly) find out which final variables were drawing information from which initial DB fields. And all we had were a bunch of scattered scripts (Python and Pig I think).
For me that translated to this task: read the scripts to trace a path of successive variable definitions in terms of other variables, eventually connecting each output variable to one or more input column name. It took me 2 long days to trace these paths by manually reading the scripts. The evening sessions were particularly mind-numbing, and involved a few glasses of wine.
Yes that isn't a thing; its a term I came up with, and I sincerely hope no one ever has to encounter such a situation.
So, someone wrote a complex data processing pipeline, that read from some tables in a DB, processed the data in small and large ways, finally producing a bunch of variables that were passed into a downstream ML model. No documentation was produced, and the developer had exited the company.
The problem: because of some high stakes issues with said pipeline's results, we needed to (quickly) find out which final variables were drawing information from which initial DB fields. And all we had were a bunch of scattered scripts (Python and Pig I think).
For me that translated to this task: read the scripts to trace a path of successive variable definitions in terms of other variables, eventually connecting each output variable to one or more input column name. It took me 2 long days to trace these paths by manually reading the scripts. The evening sessions were particularly mind-numbing, and involved a few glasses of wine.