By Ann Davis

Note: PDF version attached below.


  1. Introduction
  2. Scope
  3. Definitions
  4. Module Licensing – GPL vs. non-GPL
    1. Legal Issues
    2. Position Statement
    3. Technical Issues
      1. GPL-only Symbols
      2. Kernel Tainting
    4. Novell Specifics
  5. Source Code Maintenance – In-tree vs. External
    1. In-tree
    2. External
    3. Novell Specifics
  6. Supporting Multiple Kernel Versions
    1. Problem Statement
    2. Types of Kernel Updates
    3. Existing Solutions
      1. Requiring customers to obtain new kernel modules for each new kernel
      2. Recompiling modules “on-the-fly”
      3. Re-using modules
    4. Supporting Multiple Kernel Versions – Novell Specifics
      1. Partner Linux Driver Process (PLDP)
        1. Benefits of using the PLDP
        2. PLDP – ISV Specifics
  7. Marking Modules Supported
    1. Novell Specifics
  8. Futures
    1. Linux Foundation Driver Backport Workgroup
    2. Novell Specifics
  9. Summary

1 Introduction

As Linux becomes a commonly-deployed platform for critical business operations, demand for Linux system software solutions is growing. Linux system software products (including security solutions, system and storage management solutions, and file system clustering solutions) play key roles in the effective management of organizational processes and data.

Because system software often needs to interact closely with the underlying operating system, Linux system software products may contain kernel modules as well as user-space libraries and executables. It is particularly important that these kernel modules be reliable and robust: unlike user-space binaries, kernel modules can compromise operating system stability.

Independent software vendors (ISVs) who wish to provide kernel-space products face a number of legal and technical issues specific to kernel-space development. Assuming that a kernel module developer has licensed his source code under the GPL, he may wish to submit his module for inclusion into the mainline tree. If he decides to maintain his source outside of mainline, then he must determine how to effectively and efficiently support new kernel versions. Kernel-space software developers may also wish to build specific flags into their modules so that customers who load the modules will not be denied support from their kernel distributors.

The concerns discussed in this paper apply regardless of the Linux distributions an ISV wishes to support. However, suggested solutions may be distribution-specific. This document endeavors to highlight the basic issues, briefly describe various approaches, and provide direction for ISVs targeting SUSE Linux platforms.

2 Scope

This document is intended for developers of kernel-space software products for SUSE Linux Code 10-based operating systems (SUSE Linux 10.x, SUSE Linux Enterprise).

3 Definitions

The following terms are used throughout this document:

external kernel module – For the purposes of this document, a kernel module whose source code is not maintained in the source tree.

GPL-compatible module – Any module whose license matches one of the free software licenses specified in the linux kernel source code file linux/include/linux/module.h.

in-tree kernel module – For the purposes of this document, a kernel module whose source code is maintained in the source tree.

kABI (Kernel Application Binary Interface) – the set of symbols (including data structure layouts and versions) exported by the kernel binary and in-tree kernel modules [1].

kAPI (Kernel Application Programming Interface) – the set of exported functions, inline functions, and data members of structures that are declared in the kernel headers [1].

kABI-compatible – having the same kABI

kABI-incompatible – having different kABIs

non-GPL-compatible module – A module whose license does not match any free software license specified in the linux kernel source code file linux/include/linux/module.h.

4 Module Licensing – GPL vs. non-GPL

Note: ISVs should consult legal counsel regarding any Linux kernel module licensing issues and decisions.

4.1 Legal Issues

The Linux kernel is licensed under the GNU Public License (GPL) version 2, which includes the following statement:

“You must cause any work that you distribute or publish, that in whole or in part contains or is derived from the Program or any part thereof, to be licensed as a whole at no charge to all third parties under the terms of this License.” [2]

The above text has implications for external kernel modules. As such, developers should consult legal counsel before using any non-GPL license for their Linux kernel modules.

4.2 Position Statement

Many Linux kernel developers have signed the “Position Statement on Linux Kernel Modules” [3] to clarify their position on closed-source Linux kernel modules.

4.3 Technical Issues

4.3.1 GPL-only Symbols

The Linux kernel exports symbols using two macros: EXPORT_SYMBOL() and EXPORT_SYMBOL_GPL(). While symbols exported via EXPORT_SYMBOL() are available to any external kernel module, symbols exported via EXPORT_SYMBOL_GPL() are available only to GPL-compatible modules. The EXPORT_SYMBOL_GPL() restriction applies at module load time.

A developer defines the license which applies to his kernel module by including the “MODULE_LICENSE()” macro in the module’s source code. The string passed to this macro is then compiled into the kernel module. A module’s license status can be viewed using the
modinfo –license” command.

When a process makes a request to load a kernel module, the module loading code in the kernel performs licensing checks by:

  1. Determining the license of the module
  2. Determining the kernel symbols required by the module
  3. Checking to ensure that the module license does not conflict with how the kernel exports (EXPORT_SYMBOL() vs. EXPORT_SYMBOL_GPL()) the required symbols
  4. Loading the module only if no conflicts exist

Therefore, if a module has a non-GPL-compatible license but requires symbols which are exported via EXPORT_SYMBOL_ GPL(), the running kernel will not load the module.

4.3.2 Kernel Tainting

The Linux kernel maintains a “taint” state which is reflected by kernel messages and the /proc/sys/kernel/tainted file. A main purpose of “tainting” is to indicate various kernel events to any individual(s) debugging kernel issues. The loading of a non-GPL-compatible module is one event which always taints the kernel, as the presence of what may be a binary-only module severely impacts the success of any kernel debugging effort.

ISVs therefore should be aware that providing non-GPL-compatible modules to customers may affect the ability of those customers to obtain support for their kernels.

4.4 Novell Specifics

Novell strongly encourages all partners to GPL-license their kernel modules. As Novell’s kernels are based on the mainline source, it is to be expected that any symbols exported via EXPORT_SYMBOL_GPL in mainline also will be exported via EXPORT_SYMBOL_GPL in Novell’s products. Furthermore, Novell’s support organizations consider a customer’s kernel taint state when determining how best to support that customer. More information on kernel tainting with respect to SUSE Linux and Novell Support is available in the documents “Working with the SUSE 2.6.x Kernel Sources” [4] and “Tainted kernel” [5]. Information on Novell’s Linux support policies is available at

5 Source Code Maintenance – In-tree vs. External

ISVs with GPL-licensed kernel modules may decide to submit their module source code for inclusion in the mainline kernel source tree. The following paragraphs provide a brief discussion of the benefits and drawbacks of maintaining kernel module source code in-tree vs. externally.

5.1 In-tree

Maintaining a kernel module’s source code in-tree provides major benefits to the module developer, the kernel community, and the customer.

A module developer who has his module in-tree can expect that:

  1. The module code will receive continual review by the full Linux kernel community.
  2. The module will be included in most Linux distributions.
  3. The module will be rebuilt with each new kernel release.
  4. Any source-code compatibility problems with other parts of the kernel will be uncovered as part of the normal kernel development process.

Maintaining kernel modules in-tree helps the kernel community (including in-tree kernel module developers) to continue developing and providing a quality Linux kernel. Since a large part of the day-to-day work of kernel development is creating and submitting patches to the kernel source, it is important to ensure that a patch to one portion of the kernel does not cause build or run-time compatibility problems with any other part of the kernel (including other in-tree kernel modules). When kernel-modules are in-tree, daily builds and test runs quickly uncover any incompatibilities between patches and these modules. There is no effective means to ensure that kernel patches do not adversely affect kernel modules that are not in the kernel tree.

Maintaining kernel modules in-tree also lessens, or even obviates, the need for a stable kernel ABI/API. Since in-tree kernel modules are recompiled with every kernel build (and generally provided with each kernel update), there is no requirement for a kABI which remains stable across kernel versions. Similarly, as part of the “daily churn” of kernel builds, in-tree kernel module developers are notified about and can resolve any kAPI problems. So to the extent that kernel modules are in-tree, the kernel development community (including the in-tree module developers) can continuously improve the kernel without being constrained by having to maintain a stable binary or programmatic interface for external kernel modules.

Finally, when kernel modules are in-tree, customers benefit by having modules that have been compiled and tested with the kernel they are using, by getting kernel and kernel module updates in one package, and by being able to access and submit their own updates (if desired) directly to the source code.

5.2 External

One practical reason for maintaining kernel module source code outside of the kernel tree is that pushing code “upstream” can be a difficult and lengthy process with no guarantee that the code will be accepted. Making subsequent changes to in-tree kernel module source code can be similarly difficult. Developers must balance these issues against the long-term benefits of having kernel community support for in-tree modules.

5.3 Novell Specifics

Novell encourages external kernel module developers to submit their module source code for inclusion in Novell may be able to assist in the upstreaming process by providing information to the developer and/or contributing to pertinent discussions in kernel forums.

If an external kernel module is GPL-licensed and in the process of being upstreamed, the possibility exists for Novell to include the module in its own Linux distributions. Novell strongly favors keeping its kernels aligned with But based on business case, Novell may consider including a non-mainline kernel module in its SUSE Linux products.

6 Supporting Multiple Kernel Versions

6.1 Problem Statement

As noted in section 5.1 above, when kernel modules are in-tree, there is less need for a stable kernel application binary or programming interface. And in fact, the Linux kernel does not have a stable kAPI (or therefore, kABI) [6].

Nevertheless, many developers create, and many customers use, external kernel modules. Because these modules are not in-tree and there is not a stable kAPI/kABI, there is no guarantee that modules built for one kernel version will function correctly with an updated kernel. Developers/distributors of external kernel modules therefore must constantly update and re-distribute modules to customers or somehow create their products such that they are “resilient” to kernel updates.

6.2 Types of Kernel Updates

To effectively review different approaches of supporting multiple kernel versions, it is important to understand when and why kernel updates are released and how kernel updates may affect kernel modules.

The source changes frequently. However, most Linux distributors (including Novell) release a finite number of product versions, each based on a “snapshot” of the source at a particular time [4, 7]. In-between product versions, Linux distributors may release minor updates (bug and/or security fixes). Since each new product version includes a new kernel, a product version update is one type of kernel update. Since a bug or security fix may require updating a released product’s kernel, a maintenance/security update can be another type of kernel update.

With respect to a particular module in use on a running kernel, a kernel update (whether in a new product version or in a maintenance/security update) may be:

  • kABI-compatible with the module
  • kABI-incompatible with the module but kAPI-compatible with the module’s source code
  • kAPI-incompatible with the module’s source code

Effectively supporting multiple kernel versions requires addressing all of the above situations.

6.3 Existing Solutions

The following sections describe different methods that ISVs currently use (singly or in conjunction) to support multiple kernel versions. An advantage of these approaches is that they are reasonably distribution-independent. However, none fully addresses the need to determine kernel-to-kernel-module kABI incompatibilities at installation time while still preserving the ability to use kABI-compatible kernel modules across multiple kernel versions. And none provides a method to pro-actively push updated modules to customers. The “Novell Specifics” section below proposes a solution for these problems.

6.3.1 Method 1: Requiring customers to obtain new kernel modules for each new kernel

Many ISVs with kernel-space products provide an installation package which includes support for specific kernels, then require customers to explicitly download new module builds for any other kernel versions. This approach may simplify the ISV’s initial development. However, it does not ensure a good experience for customers, who have to pro-actively obtain and install new kernel modules after every kernel update. And if the ISV does not provide modules immediately, customers are left with non-functional systems. This customer situation is particularly frustrating given the fact that, with many kernel updates, the existing ISV modules would be fully compatible with the new kernel if only the software product included the functionality to re-use them.

6.3.2 Method 2: Compiling “on the fly”

Other ISVs install their modules’ source code onto the customer’s system, then recompile the modules “on-the-fly” when the customer starts the software product after installing a new kernel. This recompilation approach guarantees that the ISV’s kernel modules will work on new kernels which are kABI-compatible with the modules, or even just kAPI-compatible with modules’ source code. However, a drawback to this approach is the requirement that the kernel source code be installed on the customer’s system.

The Dynamic Kernel Module Support (DKMS) framework is a GPL project sponsored by Dell Computer to facilitate recompilation of kernel modules [7, 8].

6.3.3 Method 3: Re-using modules

If a new kernel is kABI-compatible with an ISV’s already-installed kernel modules, the modules can be re-used safely on the new kernel. In fact, the mainline kernel module loading code facilitates module re-use: in determining whether a module can be loaded safely, the mainline kernel checks for kABI-compatibility between the kernel module and the kernel (rather than requiring that the kernel module be compiled specifically for the running kernel.)

Given that many Linux distributors try to maintain a stable kABI through kernel security and maintenance updates, a number of ISVs use this approach to support multiple kernels with a single build of their modules. The general steps ISVs take to “re-use” modules are:

  1. At software product development time:
    1. Determine the groups of kABI-compatible kernels the product will support
    2. Build the kernel module for the first kernel in each group of kABI-compatible kernels
  2. At software product installation time:
    1. Copy all the kernel modules to a location in the software product’s directory structure (often under /opt)
  3. At software product start-up:
    1. Determine the version of the running kernel
    2. For each of the software product’s modules, use “insmod” to load the module built for the kernel version closest to but not greater than the running kernel. (Note that the kernel’s module-loading code will not load the module if it is incompatible with the kernel.)

This module re-use approach is efficient and can pro-actively address future kernel releases which are kABI-compatible with the current kernel. However, this method essentially circumvents the accepted process for loading of kernel modules (placing modules into /lib/modules, setting up module dependencies via modules.dep, and using modprobe for loading).

6.4 Novell Specifics

As noted in section 6.3 above, the previously discussed solutions allow for some distribution-independence, but none addresses the need to catch kernel-to-kernel-module incompatibilities at installation-time while still re-using kernel modules across kABI-compatible kernel versions. And none addresses the fact that ISVs need to push new modules to customers when kAPI changes require source-level modifications to the modules’ code.

Novell has implemented the Partner Linux Driver Process (PLDP) to address these problems [9, 10].

6.4.1 Partner Linux Driver Process (PLDP)

The Partner Linux Driver Process combines technology and processes to ensure, as much as possible, that customers can update to new kernels without adversely affecting the functioning of external kernel modules. Code-wise, SUSE Linux Code 10-based products support:

  1. Making kernel-to-kernel-module kABI compatibility determinations at an rpm level
  2. Re-using a kernel module if it is kABI-compatible with a new kernel
  3. Accessing multiple installation sources to update the operating system and external modules in a synchronous manner

Process-wise, the PLDP provides partners with documentation and support (based on business-case) for building, packaging, and distributing external kernel modules [9, 10]. Benefits of using the PLDP

By building software products and providing online update repositories in accordance with PLDP guidelines, ISVs can ensure that:

  1. When a customer installs the ISV’s kernel module rpm on SUSE 10.x/SLE 10, the installation will:
    1. Check all kernels installed on the system to determine which are kABI-compatible with the modules
    2. Install and configure the modules for use with each compatible kernel
  2. When a customer updates to a new SUSE 10.x/SLE 10 kernel, the update process will:
    1. determine if the new kernel is kABI-compatible with the ISV’s installed kernel module
    2. If the new kernel is compatible, install/configure the existing ISV modules for use with the new kernel
    3. If the new kernel is not compatible:
      1. If the ISV’s online repository contains a compatible kernel module rpm, prompt the customer to install the new kernel module rpm along with the new kernel
      2. If the ISV’s online repository does not contain a compatible kernel module rpm (or if the customer is not registered for the ISV’s online repository), give the customer the option to abort the kernel update

To summarize: By packaging and distributing their kernel modules in accordance with PLDP guidelines, ISVs can provide updated kernel modules to customers in an automatic manner integrated with the SUSE Linux Enterprise update process. PLDP – ISV Specifics

Full information on how to use the PLDP technology and processes is provided on the Novell Partner Linux Driver Process website [9]. However, because the PLDP is used primarily for hardware drivers, there are some specific details that ISVs should consider when building, packaging, and distributing their products in accordance with the PLDP. In general, ISVs should take the following steps to make use of the PLDP technology:

  1. Determine the software product modules to be included in a kernel module rpm. (In general, user-space libraries and executables would not be included in the kernel module rpm; see #4 below.)
  2. Determine all kernels to be supported.
  3. Determine which kernels are kABI-compatible.
  4. Create a kernel module rpm for each group of kABI-compatible kernels:
    1. Using the instructions on, create the kernel module rpm for the first kernel in each group of kABI-compatible kernels. Use the same name for all the rpms; version numbers should differ only if module source code differs.
  5. Create the online update repository containing the kernel module rpms:
    1. Using the instructions on, create an update source which contains a patch for each kernel module rpm. Use the createpatch “–patchsupplements” option to make each patch “supplement” the appropriate Novell kernel patch. This will ensure that installation of a new kernel that is kABI-incompatible with the previously released kernel will trigger installation of the appropriate kernel module rpm.
  6. Ensure that the software product’s initialization scripts load any kernel modules using the “modprobe ” command.
  7. Ensure that the software product’s installation:
    1. Uses the “rpm -q –provides” command to determine the symsets provided by the running kernel
    2. Uses the “rpm -qp –requires” command to determine which of the kernel module binary rpms on the online update repository match the running kernel
    3. Downloads the matching kernel module rpm from the online update repository
    4. Installs the kernel module rpm
    5. Using the “rug” command, registers the customer’s system for the ISV’s online update repository

When Novell releases a new kernel that is kABI-incompatible with the previously released kernel, ISVs should:

  1. Using the instructions on, rebuild the kernel module rpm for the new kernel.
  2. Using the instructions on, re-generate the online update repository with a patch containing the new kernel module rpm. The patch should “supplement” the appropriate Novell kernel update patch.

7 Marking Modules Supported

As noted in section 4 above, the mainline kernel module loading code includes functionality to set a kernel taint state for specific events such as loading of non-GPL-compatible modules. However, Linux operating system distributors also may need the ability to indicate whether a module is “supported” (i.e., will not void the user’s support agreement with the kernel distributor.) The mainline kernel module loading code does not include functionality to reflect a module’s support status, so Linux distributors often have their own approaches for building support status into kernel modules. These Linux distributors may modify the kernel module loading code in order to reflect (e.g., via an extension to the tainting mechanism) whether the kernel as a whole has any unsupported modules loaded. Including functionality to reflect module supportability status gives Linux operating system distributors the ability to establish clear and effective support policies.

7.1 Novell Specifics

Novell supports SUSE Linux Enterprise kernels in their shipping configuration. Novell does not generally support kernels which customers have recompiled or otherwise modified, such as by loading external (non-distribution) kernel modules. Before providing customer support, Novell may ask a customer to unload any external kernel modules.

However, Novell recognizes that there are situations in which customers must use non-distribution modules and cannot reasonably unload these modules to reproduce problems. As such, Novell provides developers with a way to mark their modules “supported: external” to indicate to Novell’s support teams that the module distributor has agreed to work with Novell to resolve any customer issues.

Because supporting kernels with non-distribution modules loaded may require “pulling in” resources from the module distributors, Novell requires that developers obtain explicit approval before marking their modules “supported: external”. The following must be in place:

  1. Novell must have an IHV/ISV engineer assigned to interface to the module developer/distributor (may require a Novell PartnerNet contract).
  2. The module developer/distributor must commit to provide L3 support (source-code level bug fixes) as needed to Novell customers.
  3. The module developer/distributor must be a member of TSANet.
  4. The module developer/distributor must provide support contacts to Novell.

After receiving the necessary deliverables and guarantees from the module developer, the assigned Novell engineer will provide the developer with official approval and technical instructions for how to mark his modules “supported: external”.

8 Futures

Like most ISVs, those who develop kernel-space products often wish to support many Linux distributions with a single code base. Efforts to create standard environments for developing and running Linux user-space code are making progress: The Linux Foundation Linux Standards Base (LSB) [11] has defined a standard set of APIs and provides certification tools to ensure that an application will run (unmodified) on a number of Linux distributions. Similar efforts are being made for kernel-space code, as noted below. But creating standards that apply across Linux distributions yet still allow developers to provide meaningful kernel-space functionality to customers is a difficult problem encompassing legal, political and technical issues.

8.1 Linux Foundation Driver Backport Workgroup

The Linux Foundation Driver Backport Workgroup is working toward standard, distribution-independent methods for building, packaging, and distributing/installing Linux kernel modules [12]. While the Workgroup’s focus is on facilitating backports of current upstream drivers onto older kernels, the standards being created will greatly benefit ISVs who wish to support multiple Linux distributions and kernel versions.

8.2 Novell Specifics

Novell is working to align its PLDP technology and processes with the standards being created by the Linux Foundation Driver Backport Workgroup.

As a key contributor to both the Linux Foundation LSB and Driver Backport Workgroups, Novell highly encourages ISVs to follow the work being done by these groups and make every effort to develop code in accordance with their standards.

9 Summary

ISVs who create kernel-space software must consider a number of issues that are unique to kernel-space development. Whereas Linux user-space code may be licensed (or not licensed) as desired by the developer, Linux kernel modules are subject to more stringent requirements. Maintenance-wise, Linux kernel module developers may (and are encouraged to) upstream their module source code into ISVs who do not maintain their source code in must take specific actions if they wish to keep their products compatible with kernel updates.

The decisions an ISV makes with respect to the kernel-space development issues discussed in this paper have long-term effects for the the ISV, the Linux community, and the customer.

Appendix A – Viewing Symbol Information

The following sections describe how to view the symbol versions required and exported by kernel modules and kernels.

Kernel Module Symbols

If CONFIG_MODVERSIONS is enabled, the external kernel module compilation process places required symbol versions into the “__versions” section of a compiled module’s ELF header. The easiest way to view these required symbol versions (as defined by checksum) is to use the “modprobe” command:

modprobe –dump-modversions <sample.ko>

The readelf and objdump utilities also can be used to list information in the ELF section headers.

Kernel Symbols

If CONFIG_MODVERSIONS is enabled, the kernel compilation process creates a file “Module.symvers” which lists the kernel’s exported symbols and their checksums. Depending on distribution, the Module.symvers files for different kernel flavors may be included in the kernel binary rpms and/or in separate kernel packages (e.g., kernel-syms, kernel-devel, etc.) Of course, the Module.symvers file does not include any symbols exported by external (out-of-tree) kernel modules.

Novell Specifics

In order to provide the ability to make kernel-to-kernel-module kABI compatibility checks at installation time, each SUSE Linux Code 10-based kernel binary rpm includes information (via the rpm “provides” field) about the symbol versions provided by the kernel it contains. Having a kernel rpm provide the checksum of every kernel symbol would be extremely unmanageable, so instead the rpm provides values for groups of symbols (called “symsets”).

A symset is defined by a symset file, each of which contains a list of the symbols (including checksums) belonging to that symset. Each symset filename includes a value which is a hash function of the checksums of all the symbols belonging to the symset: e.g., in the symset file “net_ethernet.494534187e344d32”, the value 494534187e344d32 is a hash of the checksums of the symbols belonging to the net_ethernet symset. The values shown in the symset filenames are the values referenced in the “provides” field of SUSE kernel rpms.

A kernel module rpm built in accordance with the Kernel Module Packages Manual contains dependencies (via the rpm “requires” field) on symset values rather than on a specific kernel version. Thus, a “Kernel Module Packages Module” rpm can be be installed on any SUSE kernel which provides the symsets required by the rpm.

Appendix B – Module Loading Process

The following outline gives a very simplified description of the mainline module loading process when CONFIG_MODVERSIONS is enabled. Terms in bold indicate user-space binaries or kernel functions.

Note: The following assumes /sbin/modprobe as the starting point. If /sbin/insmod is used instead, the process is the same from the point of the init_module system call.

  • /sbin/modprobe
    • Get full path to module from modules.* files
    • Get prerequisite modules from modules.dep, do the following for each module
      • init_module (system call, source in linux/kernel/module.c)
        • Check capabilities
        • load_module (source in linux/kernel/module.c)
          • Pull module into kernel memory
          • Do basic checks (architecture, stripped symbols, etc.)
          • Check versions and and resolve addresses for module’s required symbols:
            • Look in kernel symbol tables: gpl, future-gpl, etc.
            • Look in other modules
        • Call module’s init function

SUSE Linux uses the above mainline module loading process with an additional module support status check in the load_module function.



Novell, Inc. makes no representations or warranties with respect to the contents or use of this document and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose.

Copyright and Trademarks

© 2008 Novell, Inc. All rights reserved. Novell and the Novell logo are registered trademarks of Novell, Inc. in the United States and other countries. SUSE and the SUSE logo are trademarks of SUSE LINUX Products GmbH, a Novell business.

*Linux is a registered trademark of Linus Torvalds. All other third-party trademarks are the property of their respective owners.

(Visited 1 times, 1 visits today)
Tags: ,
Category: SUSE Linux Enterprise, Technical Solutions
This entry was posted Friday, 27 June, 2008 at 5:00 pm
You can follow any responses to this entry via RSS.


  • idhasoftseo says:

    Nice article on developing software products. By this article, I got clear explanation about Linux system software products. Thanks for sharing great and useful information with us.

  • Leave a Reply

    Your email address will not be published. Required fields are marked *