But now when you have a YAML program as the first level abstraction how do you handle results between those multi language calls ? In DSCI this is achieved via states and normal functions , and everting is just a function on general purpose programming language, in YAML you need all these magic ( awkward ) YAML syntax to mimic all those things ( pass parameters , handle global variables , process user input , handle returned parameters , etc )

If you use tools gluing them into YAML - you get YAML bloated with time . When you say those tools already having Python - excellent I would like to use those Python libs or SDK directly in my Python code instead of juggling those tools as cli or code blocks inside YAML

UPDATE: and yeah , re-read again - I guess the most of automation is done today via “CI” pipelines even when those are not meant to be CI only, like you said … anyways the rest I have said stands true for me … don’t bake your code into YAML )

The issues you had just proves that YAML based CI approach always leads to troubles with time. And yeah, I have been there, code generators for YAML. Hundreds of lines for YAML pipelines, etc ))

It allows the CI engine to determine which jobs to start, what their steps are etc.

Yep, like a said , I don’t mind to have such a configuration inside YAML, but this should NOT be pipeline code itself )

… put those in separate scripts and call them from the YAML.

This is exactly what I try to avoid, cause:

1. many people (in my experience ) even don’t bother refactoring YAML spaghetti code to separate scripts and we end up unmaintainable codebase

2. and even if one has to do such a refactoring what is the point of using YAML at all ?

All I need just a collection of tasks/jobs written on languages of choice and I don’t need YAML “programming” language at all )

PS And btw I don’t mind having a minimal amount of YAML as configuration layer and this is what is presented in DSCI, but only minimal ))

it’s just because I think in real world we have a lot of tasks where state is required or extremely beneficial, some examples on top of my head:

• creation of virtual machines with dynamic IP addresses
• creation of bug tracking system tickets with unique ticket IDs
• looking up in databases where fetched records have unique IDs

etc

Ok, try to do it on GH actions, share states between tasks/jobs for example:

tasks/task_one/task.py

#!/usr/bin/python3

update_state({
  'out1' : 'out1 value',
  'out2' : 'out2 value'
})

tasks/task_two/task.py

#!/usr/bin/python3

dict = get_state()
print(dict["out1"])
print(dict["out2"])

Or share states between jobs:

jobs/job1/task.py

#!/usr/bin/python3

update_state({
  'out1' : 'out1 value',
  'out2' : 'out2 value'
})

jobs/job2/task.py

#!/usr/bin/python3

dict = config()

print(dict["_dsci_"]["job1"]["out1"])
print(dict["_dsci_"]["job1"]["out2"])

I can’t imagine how much boilerplate code (if this ever possible ) one needs to write to achieve that on YAML based pipelines (GH Actions/ etc)

And don’t tell me about jobs artifacts ))

UPDATE:

Another good example is to run tasks conditionally, yes using old good if:

#!/usr/bin/python3

if some_condition(foo, bar): 
    run_task(
       'task1', {
          'foo' : 'foo value',
          'bar' : 'bar value'
       }
    )

The same could be quite awkward in YAML based code

Only as high level glue , main logic is just normal programming languages - see here https://github.com/melezhik/DSCI/tree/main/examples

Here is SDK for those languages:

Raku Perl Bash Python Ruby Powershell Php Golang

UPDATE: so it’s not just hitting up a language from a pipeline , it’s full reach SDK, one writes a pipeline on a language, if I get your comment correctly

GH project link - https://github.com/melezhik/DSCI