Linux Kernel: just engineering without a vision?

Monday, 15 October 07
Yesterday I bought a Microsoft webcam: it's not an exception, like most of the stuff from Microsoft I had (very little, fortunately) something to do with in my life it's a low quality product: all eye candy and little substance.

However it has a good point, that is one of the main reasons I bought it: the Lifecam NX6000 is a device compatible with the USB video class standard. In brief USB is not just a low level bus, it also defines the way some class of devices communicate with the operating system: this is the reason why we don't need to install specific mouse drivers anymore and why most USB microphones will work out of the box.

USB defines a device class for video devices, that is the UVC (USB Video Class) that new webcams are starting to implement, so my webcam can work just using the generic uvcvideo Linux driver without to require a device-specific driver, exactly like a mouse or an USB microphone.

Video For Linux 1.0 VS Video For Linux 2.0

Actually the uvcvideo diver works well with my camera, but this driver supports only the new video API of Linux, v4l2 (Video For Linux 2): this means that the majority of the software currently available can't work with my driver that dropped the support for v4l version 1.

This is not the only driver without support for v4l1 (alias for a lot of applications people are currently using). There was this fantastic idea that everything was going to be converted in short time, applications can be patched after all, and even that the support for v4l1 itself was going to be completely dropped from the kernel soon.

Check what this kernel developer is saying here...
> No, I still haven't tried Ekiga (I'm going to test it when I have the
> time), but I still need to have the webcam working with all these other
> applications.
> Do you mean that all these applications use the v4l or v4l2 interface
> incorrectly?
> Do you have plans to have v4l1 compatibility in the driver?
> I still don't know what ioctls should be in a v4l2 driver with v4l1
> compatibility using v4l_compat_translate_ioctl()
> Do you know where I can find that list?
> Do you know of a good webcam USB driver that has good v4l1 compatibility
> using v4l_compat_translate_ioctl() ?
> I saw but it has both v4l1 and v4l2
> APIs.

I don't see the point in adding V4L1 support as its due to be removed from the kernel before too long (can't remember the exact proposed kernel release but it may be 2.6.18).


I'm very impressed. We used the old v4l1 API for years and now there is the idea that this API can be dropped in short time without problems, that new drivers are not required to support it, and that there is no reason to build a working compatibility layer (there is one that can't work since the output image format is not one of the standard v4l1 formats but is MJPG instead).

Of course the v4l1 support was not dropped in 2.6.18 since it was like science fiction. The current situation basically is that if you get the wrong webcam with drivers developed by the wrong people, you have a mostly useless toy you can use just with few cam viewers using the 2.0 API and some softwares already compatible with the new API like recent versions of Egika and a few of others.

This is a point of view perfectly right for engineers, but it means to don't have any kind of vision. If you develop a kernel you should ask yourself "What I'm doing for people using this system every day" before to ask "what is the best API for video devices". There is an incredibly big failure in the Linux Kernel culture to create software that is easy to use for the people that need to use it, all is about technical stuff but what people need is supported, stable systems, where software of two years ago will continue to work without patches even if you have a modern webcam.

But Linux is getting better and better as desktop OS!

True, but are you sure it is Linux? If you take a FreeBSD install with KDE, Openoffice, Firefox, ..., it's going to be a good desktop experience exactly like a similar Linux install: most of the progresses of "Linux" on desktop are not about the kernel, they are about what is running on top of it.

Still I think the Linux kernel is a better alternative than *BSD or other Unix-like open source operating systems, for two reasons: support, and drivers.

Because Linux took more momentum something like ten years ago it started to have more developers, more money invested, more drivers, more forums, more users, more support. It is strange but continuously changing the kernel APIs (how many times a module implementing drivers for the XYZ device does not compile after few months because something changed in the API?) and a non existent stable binary API for modules, is going in the opposite direction making support for devices harder.

A Stable API for binary drivers

Ok let's try to be interested in what happens to users instead to think just as engineering: with a stable binary API what you have is the following scenario.

The user bought a new device needing a device driver in order to work, and like it happens a lot of times this driver is not already included in the kernel, or the included driver is too old to support this new USB product/vendor ID.

Without a stable API the user needs to find the source code of the kernel, download, compile (assuming it will build), insert the module.

With a stable API the user can just download the driver for Linux 2.6 and insert it: done.

On the other side the driver developer is very motivated working on the driver, all the work is invested in supporting the device better, not in upgrading and creating new tar.gz just because the API changed again (I used to develop a device driver for an USB camera and recently I tried to take a wifi driver I modified to support the monitor mode up to date and it was a pain both the times for this reason).

Note that I'm not talking about closed source drivers for now. Even if we take all GPLed a stable binary driver is already a great thing from the point of view of the user.

What about closed source vendor provided drivers?

One of the arguments against a stable binary driver is the following:
Linux is more stable than Windows because the device drivers are more stable! with a stable binary API vendors will produce closed source drivers for Linux that will crash the system every five minutes.
Is it true that Windows is instable because of drivers? After 9 years of using only Linux every day one month ago I started using Windows XP for a few hours every day: my girlfriend lives in Florence (north Italy) while I'm currently in Catania (south Italy): MSN Live Messenger is what she used in order to stay in touch with other people, and I had a Windows XP in dual boot because it was installed when I purchased this laptop and I took some space to take it in the disk (I'm a web developer and from time to time I test applications in a real Windows environment).

Result: a lot of the things I took for granted about Windows of 9 years ago are no longer true. Windows XP hardly crashes in a way that is up to the device drivers it is running: actually it mostly does not crash at all and when something goes wrong it appears to be much more related to the operating system.

I don't thing vendor device drivers are less robust than open source drivers that sometimes are written in spare time with little specification and testing. For example my wifi card with wpa_supplicant hangs from time to time, and a reboot is needed, with Windows it works perfectly. Another wifi usb dongle I use in Windows works very well supporting ad-hoc networks, while with Linux it works not very well and ad-hoc networks are not supported at all, and the list of devices working well with Linux and not working well with Windows is of course very big.

Add to this that most drivers are throw away software: a driver for a webcam will be useful for few years, the vendor needs to develop it if it want to sell the device, but isn't it a waste of time that Linux kernel developers will spend their time to write code that in few years will be nearly useless?

And closed source binary drivers will not stop everybody from producing GPL drivers: for important parts of the system like for example the USB subsystem or the IDE controller, and even for small devices like webcams where the vendor driver sucks probably a GPL driver will be developed like it happens today.

So I'm starting to belive that the Linux Kernel is not going in the right direction from the point of view of the end user, and I guess the only thing I can do for now is to try to spread this idea trying to start some kind of conversation among interested people.

You can vote this article on reddit.

p.s. this blog used to be written in Italian, but this article is the first of the Big Switch to English, so feel free to come back if you are interested in this stuff and sorry to my old readers that can't read english very well...
Posted at 17:20:06 | permalink | 13 comments | print
Do you like this article?
Subscribe to the RSS feed of this blog or use the newsletter service in order to receive a notification every time there is something of new to read here.

Note: you'll not see this box again if you are a usual reader.


she writes:
15 Oct 07, 18:51:39
Although I dont agree totally, I understand and support your basic point.
We can take another example, the OSS vs ALSA madness that was here... my gripe with it all is that Linus does not provide a clear vision that goes out of his kernel. And obviously neither inside.

With that I mean, one way to FOLLOW. People can still strive away from this for whatever reason but at least you KNOW that this one way will work.
It also happened with the change to udev. udev complicated so much... now the same happens with PAM and HAL ... I think sometimes development just happens to make things more complicated.
Another example, I know how to configure kdm and gdm, but for me its just easier to do "rinit gnome" and this will start gnome via .xinitrc... or "rinit flux" and it will start fluxbox etc.. the total script is around 20 lines of code, and without args it will just startx...
simple and effective without the need for those fancy add-ons like kdm..

In actuality though, i want one thing to WORK, and work in a GOOD way. And this is where I disagree with you in this one aspect:

IF something will be BETTER, than I do NOT mind at all if legacy is thrown away.
(I'd just wish that people would throw away the ugly FHS, and that Linus would, instead of flaming gnome or whatever-i-dont-care, give a clear vision of how things SHOULD look. It would help a LOT, because after all, Linux as a whole is effective, but a TOTAL MESS)
antirez writes:
15 Oct 07, 19:05:22
@she: basically it's exactly what I mean, the benevolent dictator is not working for subsystems of the kernel, and of course can't work at all for the "userspace madness", but at least the userspace complexity and inconsistency is handled by distributions and GUIs in some way... instead for the problems about the kernel there is almost no solution.

About your second point where you disagree, my idea is that of course it's not possible to go slow just to support legacy things, but:

a) probably some API was designed in a bad way if in order to make it better you need to totally break it and rewrite it from scratch.

b) many times, like for the V4L1 problem, it's possible to write a compatibility layer without too much efforts, why this isn't done at all? It's a shame. Also I don't think that changing the interface between kernel and modules is required in order for the kernel to work well. It should be like that for kernel 2.4 the interface must be taken stable, than 2.6 can change it and so on. It makes a lot of sense to bring a bit of API stability, there is always room to improve in the next release and if the API is well designed there is no reason at all to break things.

Thanks for commenting.
chuck writes:
15 Oct 07, 21:03:00
The term for a "binary API" is ABI, and the official policy of Linux is that there is no stable ABI, period.

That Linux can't manage to keep even the source-level API stable, or keep the ABI stable between patchlevel releases is a serious problem. OpenSolaris looks more tempting every day, and as more drivers are open-sourced, the value proposition of sticking with Linux is starting to diminish.
15 Oct 07, 21:39:38
Remember to add hardware that doesn't work with Linux to the Linux Incompatibility List, here:

Paul writes:
15 Oct 07, 23:51:07
Don't forget about the tremendous work done by gspcaview/spcaview. Also its important to realize that driver development for these devices are done at a chip specific level, and not by the manufactures that assemble the final board arrangements which come in a far greater variety, requiring more overall testing. Still I think its wrong for management at these device manufacturing companies to limit the distribution of in-house code to run these devices. You have to believe that the developers that work on these devices have code in-house that runs on GNU/Linux, doing development on such systems out of convenience and just not releasing as gpl due to upper management. If thats the case the issue just needs greater awareness; I don't think a manager has a reason to even think of such a release concept.
liquidat writes:
16 Oct 07, 09:36:35
Hm, I think you don't take into account why things are like that atm., and in which directions they are changing.

Of course, the development around the Video4Linux drivers is a mess and there was something really wrong in the development. The result is that it will take some time until all this smoothes out. Sad, but true.

However, in almost all other regions there was massive development in the opposite direction just recently:
* the WLAN subsystem has been massively reworked, resulting in a stable subsystem in contrast to six different subsystems.
* the MMC subsystem has been massively reworked to now support standards instead of just patching together the support for different devices

If you go some time back you find other, major and important developments: with D-BUS Linux finally gained a system wide IPC and HAL and udev improved the device handling and the possibility to react to different devices and different device quirks massively (she: when you think udev and HAL complicated anything there is something clearly wrong!)

Also, future developments are clearly outlined: the next major subsystem rework will be the one of the graphics subsystem. This is again not to break anything on purpose but to bring some sanity into the current solution which has many problems and blocks real performance on X.

Many of these changes are necessary out of historic reasons: now there is finally the money and time (=development power) to work on these subsystems in a clear an sane way instead of just patch-as-you-go.
Of course the switch should usually be like the WLAN switch (just delete old drivers if there are no regressions with the new ones in any way) and not like the V4L switch from 1 to 2.

Also, keep in mind that these switches of the happen also for another reason: performance improvements. Many of the old APIs have only limited scalability and performance, and new ones are necessary to really use the hardware.
Who wants to have a suboptimal solution?

About the question of a general vision: the vision is to let all hardware work in the best way. And to let the subsystem people do what they think is best. What kind of vision would you like to have? Keep in mind that this is not a single company with a product with a specific target but a development project with literally hundreds of targets (server, embedded, desktop, mars rover, etc.).

Also, about the stable API: this is already possible with quite little effort (userspace driver api) - but the question is if that is really necessary. If the subsystem development would have been made in a sane way it could have worked :/
Also, for a stable long term one-size-fits-it-all stable API you should keep in mind what the consequences would be like:

So: I think the best way would be to address the subsystem people of Video4Linux and point out that something went horribly wrong. Just the fact that the new Flash 9 (released not too long ago) choose V4L1 instead of V4L2 already showed that there are some big misconceptions and problems in that area, and that the subsystem people did something horrible wrong there.
They should have at least talked to the users of the API (the application developers) so that V4L2 should have been supported at release date by all important apps. Also, the porting of the driver had too many problems apparently and that should have been smoothed out before the final release.
But that has not that much to do with a stable API but with coordination between developers and "users".
wageslave writes:
16 Oct 07, 14:05:09
Sometimes, you get what you pay for.
davidonzo writes:
16 Oct 07, 14:33:36
1 - Can I ask you where you bought it? I'm looking for a good webcam that works easly with my distro (ubuntu feisty).

2 - Linux is improved its desktop usability because desktop application work on this way.
I never see new linux kernel with "real" desktop oriented features.
Most of the job to make a real desktop oriented distribution is made by the singles distro team.

The linux kernel included in Ubuntu is never the same released by the OSDL.

The reason? Ask to Linus :)
But, maybe to release a single kernel for desktop and server machine make some decision not so easy to take.

3 - Translate your "mobile edition" (words "commenti" and "successivi").

4 - Thanks to switch this blog in english. Another way to improve it!
Ev writes:
16 Oct 07, 16:05:11
If you are asking for stable kernel APIs, you are missing the point of Linux: it is free software. The source code for drivers of your camera, or for anything for that matter, should be available for everybody to see and change, therefore they are supposed to be updated in the same fluent and promptly manner as the kernel itself.

What you are asking for is a commercial software model except you conveniently are not willing to pay for it. You misinterpreted the meaning of "free".
spaceman writes:
16 Oct 07, 17:58:08
"What kind of vision would you like to have? Keep in mind that this is not a single company with a product with a specific target but a development project with literally hundreds of targets (server, embedded, desktop, mars rover, etc.)."

Mars rover does not use a linux kernel at all : watchdog subsystem is incompatible with a correct management of any kind of malfunction . In the 1997 a similar system completely fails , 'cause of a bug that caused the system to continuosily reboot. The new "sofware politics" of mars rovers is to use a microkernel structure that is able to make diagnostics against the misfunctional code or to isolate the entire subsystem , in the worst case.
Drew Vogel writes:
16 Oct 07, 19:56:06
A better thing for the user (than even downloading a stable binary driver) is Debian's module-assistant that downloads, compiles, and inserts modules for you, relying on the already stable Depends system for handling API incompatibilities. Not only is it a single step but it also allows the drivers to get better over time.
liquidat writes:
18 Oct 07, 23:34:50
spceman: Yes, you're right, the Linux part of Mars rover was actually a bit different:
"On MER, Linux is being used for high-level science planning and for low-level command sequencing, visualization and simulation."

But that isn't the point here.
Alex writes:
25 May 09, 11:02:16
Even single-company targeted *ndows has much more problems with the drivers and "stable" API than you can imagine. Some examples:

1. *rectX 9/10 has to bring more than 100 megabytes of compatibility DLLs by now, and the number will grow. A way to fight API/ABI changes, for sure. But there is no way to avoid them :)

2. "stable" API broken *rectSound3D support in *sta and upcoming *indows Seven bullcrap. And there was even no compatibility layer provided. The same as your case.

3. The same "stable" API broken plenty of application compatibility in the same bullcrap.

4. *ndows Installer is updated from 1.0 to 2.0 to 3.0 to 3.1 breaking many installers with each update.

5. Plugging in Creative webcam into XP SP2 or higher results in a stable blue screen crash. SP1 works ok. The "stable" API in action, yeah. Sometimes it is better to break API/ABI than to cope with probable bugs.

6. Try to run most of the XP drivers on *ista bullcrap. You'll be surprised with the ABI stability. And this is included *ista did not changed much in the kernel. Kernel second minor changes (i.e. 2.6.18 to 2.6.19) are bigger than that.

7. Try to run some XP sound drivers on 2003 kernel. You'll be surprised, too. And that's the same kernel.

8. Much much more can be mentioned.

"Stable" API/ABI is a dream or even a miracle. It can be keeped as long as the product has no reasons to get significantly better. When it has, API/ABI is changed.
comments closed