Forum Discussion

Norbert_Kisiel's avatar
Norbert_Kisiel
Icon for Quantexa Team rankQuantexa Team
1 hour ago
Solved

Best practice for setting runIds for ingest and batch resolver

Hi team, been a while since looking at this.

We have three data sources running at different times, followed by BR. What's the best practice for setting runId for these via an automated process? Recognising that the BR needs to reference each folder where the consolidated output is.

6 Replies

  • Rafal_Bartosz's avatar
    Rafal_Bartosz
    Icon for Quantexa Team rankQuantexa Team
    1 hour ago

    Setting the data ingest RunIDs for QBR to read for each of the data sources? (I want to clarify where you want to be setting the run ids)

    • Norbert_Kisiel's avatar
      Norbert_Kisiel
      Icon for Quantexa Team rankQuantexa Team
      1 hour ago

      The question is asking best practice for both the data ingest and the QBR.

      Your response answers the best practice for QBR, thanks!

      For the  data-ingest, we were planning to set the runId to the timestamp the job commences, essentially by supplying as a command line property to the job.

      • Rafal_Bartosz's avatar
        Rafal_Bartosz
        Icon for Quantexa Team rankQuantexa Team
        1 hour ago

        The best practice for setting a RunID is to align it to the ID of your run in your scheduler if that's possible, e.g. an airflow run id. Passing this in via a command line property is good.

  • William_Logue's avatar
    William_Logue
    Icon for Quantexa Team rankQuantexa Team
    1 hour ago

    I think it's just scoring that needs the run IDs to be set manually now, QSL also pulls it from the metadata