/tech/ - Soyence and Technology

Gentoo in general is just disappointingly underfeatured. Soft USE flags need to be a thing but aren't and I can't see why. The firmware package is a retarded 1.7GB pack with thousands of devices instead of being scripted to install only the needed ones. Python has no place running major system components, including the package manage. Slow memory hog POS bloatoo rasin poop nigger

Reppie my thread gentooGODs

>>29776
Also per package env flags are really annoying.

As far as I know there isn't a complete solution to this. You could do something like
>emerge package
>wrapper figures out what dependencies/use flags should be used
>sshes into the server, builds the packages in a chroot environment that matches laptop's environment
>server builds them and drops them in an http fileserver
>laptop pulls them
This whole thing wrapped in a script.

ATP I want to try making my own version of portage. I've chatted with some of the gentoo devs and they seem much more competent than me so my guess is that there is some unforeseen complexity I don't know about. If I do I'm going to write it in C and I'm calling it sortage

>>29891
I'll make an outline of how it will work later. It will still work pretty much the same and use current ebuilds doe

bvmperino, did you make something op?

>>30117
I've been really busy but I will make an outline sqqn

>>30120
Alright I'll wait

Bump summer break has started so I will get on this

So far I've only really made notes with a few ideas I'm tossing around. TLDR I want portage but modularized to the point that every stage of the process is able to be handed off to another machine over network. The discrete steps are:
1) Evaluate dependencies and their USE flags
2) Downloading/Preparing the source
3) Pre-compilation checks and compilation/spawning distcc jobs
4) Installing the completed binary and/or saving for use as a binhost

The idea is that a client can outsource any point or points in the list to another host machine. I think the easiest example of a host would be something like a home server. It can act as a compile coordinator, an ebuild repo, and a provider of profiles, release archives/source repos, or binaries over LAN.

I think the rest comes pretty naturally.

If the client wants to outsource step 1, it packs up its list of ebuild repos, accept keywords, its active profile name, and USE flags in a tarball, then sends it to a host who evaluates the upgrade/install with a copy of the requested repos and copy of the target profile which is most likely synced with a cron job every day or so. The resulting list of packages can either be sent back, passed on to another machine, or directly fed into step 2.

Step 2 works similar to how it does now but with the addition of local "sourcehosts", that can fetch and stores source code for an entire network when sent a request. I would prefer if source repos were used in place of plain archives on hosts so any machine requesting an update could be given only the relevant patches to apply to a local archive. To my knowledge, portage doesn't work this way as opposed to git and I don't know why. Hosts should also be able to optionally send entire plain archives in the case that a client requests a package it does not have any version of when commit history is unnecessary. Public sourcehosts that local ones grab from should also work this way.

Step 3 is simple, just take source code and either compile or distribute it with projects like distcc or icecc.

Step 4 is also simple. If you are a host, you send the binary to the client and optionally store it for use with other machines on the network. If you are a client with a binary, you just install it normally.

One of the benefits of a system like this is that you can create incredibly small Gentoo systems. If you outsource all steps, you eliminate the need for code for dep evaluation, storing the current profile and ebuilds, and a compiler toolchain. Another is the bandwidth saved with centralizing things locally.

>>31431
I have a couple more ideas I might flesh out tomorrow like soft USE flags and using torrents for public or massive local repos. And also ways to cut down on network traffic from copying ebuild repos to machines that evaluate their own dep trees. For now I sleep

>>31432
Pretty much soft USE flags are flags that can be overridden by ebuild similar how the profile can override package.use or make.conf. It precludes a lot of explicit per package exceptions

>>31431
It's come to my attention that *-9999 packages not only clone an entire branch every time, but they do it every single update regardless of whether any new commits are made unless app-portage/smart-live-rebuild is installed. I am 100% certain any sane rewrite of portage will depend on git for version management. In my opinion git should be distributed via bittorrent but creating such a thing is way out of my scope

>>31432
>And also ways to cut down on network traffic from copying ebuild repos to machines that evaluate their own dep trees
Ignore this, it's retarded. I was thinking clients only need the ebuilds of packages currently on the system, but that pretty much requires the host to evaluate the deps to find out which to send anyway, so it might as well just return package names. If a client wants to evaluate deps it should have a full copy of the repo. I could see the host trimming it down for clients based on licensing or USE flags but then that also needs version control to be efficient

>>30121
>>30117
I love you random /tech/GOD please come home…

>>31494
>>31484
>>31431
As important version control is to many aspects of what I'm trying to do, maybe taking a look at the source of git and its alternatives is worth my time. First I want to get it working with as few custom components as possible, but bandwidth optimized, master/slave decentralized git sounds sweet for many other uses if it doesn't already exist

Uphunk

It just occurred to me that USE flags should just be patchsets with extra metadata for dependencies

>>31723
That could also be good for optional user patches that wouldn't just be applied blindly if in /etc/portage/patches

>>31723
Fetching source should be USE flag aware so only the relevant code is pulled

>>31494
The entire ebuild tree should be handled with version control too. Of course why redownload the entire thing when 99% of it doesn't change update to update? You don't even have to replace individual update ebuild files, you could patch those just as you would source. Only include the entire oldest version of each package and then provide a set of patches for each subsequent ebuild. Most updates will not even have additional patches.
The idea that USE flags are just patchsets means could add external patch repos. I could imagine this would be especially useful in suckless adjacent projects. The repo would supply patches that add USE flags to the relevant ebuild files and the patches that are applied by those USE flags in the source itself

Repie my thread I need feedback

>>29775 (OP)
use arch instead

use debian instead

>>31933
See my thread here

>>31934
Aryan but no

someone needs to make convincing benchmarks for gentoo adding literally any benefit to performance, memory usage, etc etc. ive used it for 2 years because im a fatty who wanted to use what everyone said was the hardest distro, but after switching to artix im very happy just being to fucking download something and have it be ready in under a minute. it makes me wonder if the gorillion years of compiling was worth it at all

>>31937
There's a bunch of large binaries packages for amd64 that you can use if you are a poorfag. Otherwise buy a gigaputer or build a cluster poor ass nigga, kek!. Most of the RAM benefits come from USE flags and using minimalist packages and not compiler flags doe. But as for performance, I think cachy has shown they do indeed do something

>>31938
ev&doe on a 100k dollar computer emerge will still never come close to the speed of just downloading something. like i said, i used it for 2 years and my computer is fine. if youre going to use binhost you might as well stop coping and just use a binary distro, since the selection of gentoo packages is very limited compared to something like artix

>>31939
The speed of portage is what this thread is about. Part of my goal is to make local network binhosts more viable. Also reduce network traffic and replace python.
Also another angle many GentooGODS take is that if your compile times are too long then you should just compile less code. And if you are compiling as little code as you can then maybe your compiler is bloated and you should just write code that doesn't need a niggerbillion optimization steps

>>31940
the speed of emerge is bottlenecked entirely by compile times, how much could you possibly save by making portage faster? like 10 seconds per invocation?

>Also another angle many GentooGODS take is that if your compile times are too long then you should just compile less code.

i look forward to someone "just" writing a browser that has actual extension support then because so far it hasnt happened

>I want a way to have the server emerge packages on demand with custom flags from the target machine.

bash wrapper that sends the command to the binhost over ssh

>>31941
For individuals, I don't see private binhosts speeding up ondemand emerges that much, but I could imagine a coordinated group of friends or large sysadmins using it. A script would be much too complicated and hacky for what I want to do, I already considered it. Plus the whole version control paradigm is too interesting for me to pass up on. I think it could be applied to the broader internet as well but I think this would be a good proof of concept

>>31941
I think you might enjoy picking apart my earlier babblings. I'll make a more comprehensive outline later and someone can feed it into Claude if they want geg

Ultimately I wish for a compiler that only refactors code affected by patches. I don't think this is fool proof because of the halting problem but it would be cool to see an attempt. And no this is not ccache

>>31944
To halting problem also applies to normal compilers so that's moot