norden.social

0 posts0 participants0 posts today

Christian MeestersLast year at the Open Science Retreat (<a href="https://fediscience.org/tags/OSR24NL" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#OSR24NL</a>) I have been introduced to nanopubs by <a href="https://social.edu.nl/@egonw" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@egonw</a> and created my first nanopub declaring citations for a paper using CiTOs (citation ontologies).Now, travelling to <a href="https://fediscience.org/tags/OSR25CH" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#OSR25CH</a>, due to issues with the train network foreseen with plenty of time, I used the opportunity and created a new one ( <a href="https://w3id.org/np/RAicq7k9QHX8EG8ho7Baib9GxHsU18O0tyFCZ4tbbGGPA" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://w3id.org/np/RAicq7k9QHX8EG8ho7Baib9GxHsU18O0tyFCZ4tbbGGPA</a> ) for my latest publication on teaching <a href="https://fediscience.org/tags/Snakemake" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Snakemake</a> on <a href="https://fediscience.org/tags/HPC" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#HPC</a> systems.The teaching material is — again — in desperate need of additions and overhaul, but that is for another day.<a href="https://fediscience.org/tags/OpenScience" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#OpenScience</a> <a href="https://fediscience.org/tags/ReproducibleResearch" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ReproducibleResearch</a> <a href="https://fediscience.org/tags/ReproducibleResearch" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ReproducibleResearch</a> <a href="https://fediscience.org/tags/AcademicChatter" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AcademicChatter</a>

Johannes KösterThis feels pretty huge! The <a href="https://fosstodon.org/tags/Snakemake" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Snakemake</a> docs now have an LLM powered chatbot (provided for free by <a href="https://mastodon.social/@gurubaseio" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@gurubaseio</a>). It can be seen as a replacement for the normal doc search. This can help a lot, in particular when you are unsure about the keywords to use for a doc search.

Johannes KösterThere have been several new <a href="https://fosstodon.org/tags/Snakemake" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Snakemake</a> releases in the last days. Apart from various fixes, the most important new feature is the ability to annotate input file access patterns. This enables storage backends tailor their provisioning mechanism for each file: <a href="https://snakemake.readthedocs.io/en/stable/snakefiles/storage.html#access-pattern-annotation" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://snakemake.readthedocs.io/en/stable/snakefiles/storage.html#access-pattern-annotation</a>

Pierre Lindenbaumdoes "snakemake --config config=in.yml -n --export-cwl "out.cwl works ? I don't have any ouput ( snakemake 9.1.3)<a href="https://genomic.social/tags/snakemake" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#snakemake</a> <a href="https://genomic.social/tags/cwl" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#cwl</a> <a href="https://genomic.social/tags/workflow" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#workflow</a>

Yann Büchau :nixos:> Removing output files of failed job product2animation since they might be corrupted😠😠😠Thanks for nothing <a href="https://fosstodon.org/tags/SnakeMake" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#SnakeMake</a> for deleting a very precious output file, resulting from an hours/days-long process because it *might* 🪩 be corrupted 🤦I though we were past this nonsense of build-systems deleting files at will, especially with a science-focused project such as snakemake!?

Yann Büchau :nixos:Ugh, <a href="https://fosstodon.org/tags/SnakeMake" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#SnakeMake</a> apparently has a hard requirement of all input (and previously-made output-) files to be present on-disk. Can't even do a dry-run `snakemake -n`. 😩<a href="https://fosstodon.org/tags/rdm" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#rdm</a> <a href="https://fosstodon.org/tags/reproducibleresearch" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#reproducibleresearch</a>

Yann Büchau :nixos:I want a build system that:- is as powerful and flexible as <a href="https://fosstodon.org/tags/SCons" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#SCons</a> - as readable and concise as <a href="https://fosstodon.org/tags/SnakeMake" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#SnakeMake</a> - has a fricking progress bar+ETA - is :datalad: <a href="https://fosstodon.org/tags/datalad" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#datalad</a> / :gitannex: <a href="https://fosstodon.org/tags/gitannex" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#gitannex</a> agnostic (knows that files can be fetched from elsewhere - remembers how long building things takes - balances that to decide if rebuilding locally instead of fetching gigabytes via slow internet is favorable - integrates well with :nixos: <a href="https://fosstodon.org/tags/nix" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#nix</a> for reproducibility<a href="https://fosstodon.org/tags/rdm" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#rdm</a> <a href="https://fosstodon.org/tags/dataAnalysis" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#dataAnalysis</a>

Yann Büchau :nixos:<a href="https://fosstodon.org/tags/SnakeMake" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#SnakeMake</a> has some nasty caching going on, where it sometimes reuses old versions of code you have long overwritten. Removing the .snakemake/ folder solves it. Also this behavior that if there's an error in your notebook, Snakemake will just *not* save the processed notebook so you can't actually have a look at the error. Makes it kinda annoying to debug.

Yann Büchau :nixos:It is beyond me why build systems like <a href="https://fosstodon.org/tags/make" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#make</a> <a href="https://fosstodon.org/tags/scons" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#scons</a> <a href="https://fosstodon.org/tags/snakemake" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#snakemake</a> et al. all don't have a flag like `--pattern '*2022-*'` that will make only those files it knows how to make matching a given pattern.(using shell brace expansion and/or globbing is risky and doesn't work fully if not all the files exist yet)

Christian MeestersToday is the day of closed pull request for <a href="https://fediscience.org/tags/Snakemake" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Snakemake</a>. The <a href="https://fediscience.org/tags/SnakemakeHackathon2025" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#SnakemakeHackathon2025</a> participants worked at full speed!We decided to write a white-paper summarizing our achievements rather than posting individual things. Suffice to say, that also the documentation made a great leap towards better readability!<a href="https://fediscience.org/tags/OpenScience" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#OpenScience</a> <a href="https://fediscience.org/tags/ReproducibleComputing" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ReproducibleComputing</a> <a href="https://fediscience.org/tags/ReproducibleResearch" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ReproducibleResearch</a>

Johannes KösterThe second day of the <a href="https://fosstodon.org/tags/SnakemakeHackathon2025" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#SnakemakeHackathon2025</a> is over. Today I have merged 11 fantastic pull requests in the in main snakemake repo alone. And we have our first <a href="https://fosstodon.org/tags/Snakemake" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Snakemake</a> release during the hackathon, adding integration with xonsh for scripting and helpers for extracting params from input files.

Johannes Köster<a href="https://fosstodon.org/tags/Snakemake" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Snakemake</a> has an update best practices guide: <a href="https://snakemake.readthedocs.io/en/stable/snakefiles/best_practices.html" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://snakemake.readthedocs.io/en/stable/snakefiles/best_practices.html</a>

Johannes Köster<a href="https://fosstodon.org/tags/Snakemake" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Snakemake</a> 8.29 has been released. It provides various improvements to the integrated reporting, as well as many small bug fixes: <a href="https://snakemake.readthedocs.io/en/stable/project_info/history.html" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://snakemake.readthedocs.io/en/stable/project_info/history.html</a>

Christian MeestersThere are many HPC admins who prohibit using considerable CPU time on login nodes. This is understandable.I want to take this opportunity to provide a data point. My student has measured the accumulated CPU time (user + system) for a 9 h (precise: 33343 s) run of a Snakemake workflow. It was 225 s or about 0.67 % - including jobs which were carried out on this login node, e.g. `mv`, `ln` or download of data.There is certainly room for improvements. There will ever be room for improvements.But my dear fellow admins: Running Snakemake on login nodes as a shepherd of jobs, will impair nobodies work.Over time, I will certainly gather more and different statistics. And will invest time in necessary improvements. Regarding CPU time for checking job status, however, I believe to have demonstrated, that this is a pretty high hanging fruit.<a href="https://fediscience.org/tags/HPC" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#HPC</a> <a href="https://fediscience.org/tags/Snakemake" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Snakemake</a> <a href="https://fediscience.org/tags/reproduciblecomputing" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#reproduciblecomputing</a>

Christian MeestersTomorrow? Trip to Dresden. Towards a course, I give about reproducible computing with <a href="https://fediscience.org/tags/Snakemake" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Snakemake</a> on <a href="https://fediscience.org/tags/HPC" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#HPC</a> clusters.I will take a few <a href="https://fediscience.org/tags/FediScience" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#FediScience</a> flyers with me. Just in case someone is interested in <a href="https://fediscience.org/tags/OpenScience" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#OpenScience</a> communication, here on <a href="https://fediscience.org/tags/Mastodon" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Mastodon</a>. 😉 <a href="https://fediscience.org/tags/Wisskomm" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Wisskomm</a> <a href="https://fediscience.org/tags/sciencecommunication" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#sciencecommunication</a>

Peter Cock<a href="https://fosstodon.org/@johanneskoester" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@johanneskoester</a> Random <a href="https://fediscience.org/tags/snakemake" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#snakemake</a> question for you - should with work with SLURM where the default shell is Dash not Bash?I have some code and workflows which works on SLURM cluster A where /bin/sh is an alias for bash, but fail on SLURM cluster B where /bin/sh is an alias for dash:WorkflowError: SLURM sbatch failed. The error message was /bin/sh: 1: Syntax error: "(" unexpectedI don't have any brackets in my workflows, but there are some in the data files…

Christian MeestersYesterday, I learned that we have only 16 - in letters: sixteen(!) - people attending my course in Dresden, so far.So, if you are interested in creating sustainable data analysis workflows using <a href="https://fediscience.org/tags/Snakemake" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Snakemake</a> on <a href="https://fediscience.org/tags/HPC" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#HPC</a> systems, this course might be something for you:On the 26th and 27th, I will be teaching a 2-day course in <a href="https://fediscience.org/tags/Saxony" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Saxony</a>, <a href="https://fediscience.org/tags/Dresden" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Dresden</a>. See <a href="https://tu-dresden.de/zih/hochleistungsrechnen/nhr-training/bd-hpda/snakemake" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://tu-dresden.de/zih/hochleistungsrechnen/nhr-training/bd-hpda/snakemake</a> for details and registration.<a href="https://fediscience.org/tags/RSE" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#RSE</a> <a href="https://fediscience.org/tags/ReproducibleResearch" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ReproducibleResearch</a> <a href="https://fediscience.org/tags/OpenScience" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#OpenScience</a>

Christian MeestersWhen reading this call (<a href="https://mast.hpc.social/@sneuwirth/113945330638974358" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://mast.hpc.social/@sneuwirth/113945330638974358</a>) from <a href="https://mast.hpc.social/@sneuwirth" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@sneuwirth</a> I think, I could tell a number of tales from my experience with developing the <a href="https://fediscience.org/tags/Slurm" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Slurm</a> plugin for <a href="https://fediscience.org/tags/Snakemake" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Snakemake</a> (and other software over the years).Particularly, issue reports telling what peculiarities fellow admins thought of (e.g. "we are not allowed to do this" and "we not that". "We have this setting" and "we that".) To develop <a href="https://fediscience.org/tags/HPC" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#HPC</a> software is a challenge in its own right. To support workflow software seems to develop into a nice challenge.Could certainly deliver a funny few minutes. 😉 Perhaps I should collect anecdotes and write an article sometime? 🤔

Johannes KösterThe <a href="https://fosstodon.org/tags/snakemake" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#snakemake</a> workflow catalog has since recently >300 registered workflows with standardized and unified usage instructions . Furthermore, Michael Jahn from Emmanuelle Charpentier's lab is now the new maintainer of the catalog: <a href="https://snakemake.github.io/snakemake-workflow-catalog" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://snakemake.github.io/snakemake-workflow-catalog</a>

Christian MeestersAgain, we have a new patch-level release of the <a href="https://fediscience.org/tags/Snakemake" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Snakemake</a> executor for <a href="https://fediscience.org/tags/SLURM" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#SLURM</a> on <a href="https://fediscience.org/tags/HPC" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#HPC</a> systems.Turned out, that some clusters do not allow account checking with `sacctmgr` (which was there for historic reasons), hence, we now have a fall-back to `sshare`. See <a href="https://github.com/snakemake/snakemake-executor-plugin-slurm/releases/tag/v0.12.1" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://github.com/snakemake/snakemake-executor-plugin-slurm/releases/tag/v0.12.1</a> for details.<a href="https://fediscience.org/tags/OpenScience" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#OpenScience</a> <a href="https://fediscience.org/tags/ReproducibleResearch" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ReproducibleResearch</a> <a href="https://fediscience.org/tags/ReproducibleComputing" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ReproducibleComputing</a>

Recent searches

Search options

Administered by:

Server stats:

#snakemake