Cluster Pools got marked read only, OSDs are near full.

IMPORTANT NOTE: With SES 6 and later it is recommended to activate the balancer module instead of making manual OSD weight changes, for details see the SES 7.1 online documentation .

OSDs should never be full in theory and administrators should monitor how full OSDs are with "ceph osd df tree". If OSDs are approaching 80% full, it’s time for the administrator to take action to prevent OSDs from filling up.  Action can include re-weighting the OSDs in question and or adding more OSDs to the cluster.

Ceph has several parameters to help notify the administrator when OSDs are filling up:

# ceph osd dump | grep ratio
full_ratio 0.95
backfillfull_ratio 0.9
nearfull_ratio 0.85

By default, when OSDs reach 85% capacity, nearfull_ratio warning is triggered.

By default when OSDs reach 90% capacity, backfillfull_ratio warning is triggered.  At this point the cluster will deny backfilling to the OSD in question.

By default when OSDs reach 95% capacity, full_ratio is triggered, all PGs (Placement Groups) on the OSDs in question will be marked Read Only, as well as all pools which are associated with the PGs on the OSD. The cluster is marked Read Only, to prevent corruption from occurring.    

It is good practice to set the "noout" flag when in this situation to avoid re-balancing if OSDs go down.

How to set/unset noout:

ceph osd set noout
ceph osd unset noout

To get the cluster out of this state, data needs to be pushed away or removed from the OSDs in question. In the below example it is a single OSD in question (osd.52), but there could be many OSDs that are marked full.  

To push data away from the osd, run:

ceph osd reweight $osdID $Weight

Where $osdID is the OSD number "52" and $Weight is a value below 1, for example:

ceph osd reweight 52 .85

The first objective is to get the OSDs that are full below 95% capacity, so the OSDs /pool(s)/cluster is not marked Read Only.  It is possible to achieve this goal with a $Weight of .95 in some cases, and in other cases, it may require a lower value, .90, .85, .80, etc.

The second objective is to get the OSDs in question first below 90%, then below 85% of capacity.  This is achieved by continuing to monitor the cluster and re-weighting down the OSDs in question.  This process can take a few hours to complete.

Keep in mind that when data is being pushed away or removed from an OSD, that data is being placed on another OSD in the cluster.  This action can in turn cause other OSDs to fill up as well.  The administrator will need to monitor ALL OSDs with "ceph osd df tree" to ensure that proper action is taken. 

In some cases it will be necessary to change the following settings temporarily: 

ceph osd set-nearfull-ratio <ratio>
ceph osd set-backfillfull-ratio <ratio>
ceph osd set-full-ratio <ratio>

"ceph osd set-full-ratio .96" will change the "full_ratio" to 96% and remove the Read Only flag on OSDs which are 95% -96% full. If OSDs are 96% full it's possible to set "ceph osd set-full-ratio .97", however, do NOT set this value too high.

"ceph osd set-backfillfull-ratio 91" will change the "backfillfull_ratio" to 91% and allow backfill to occur on OSDs which are 90-91% full.  This setting is helpful when there are multiple OSDs which are full. 

In some cases, it will appear that the cluster is trying to add data to the OSDs before the cluster will start pushing data away from the OSDs in question.  

Once OSDs are below 95% capacity (full-ratio) or below the 90% capacity (backfillfull-ratio), set the setting sback to default:

ceph osd set-nearfull-ratio .85
ceph osd set-backfillfull-ratio .90
ceph osd set-full-ratio .95

This will ensure that there is breathing room should any OSDs get marked full again at some point in time.  

If the Administrator is confident the issue addressed and it is safe to re-weight OSDs back up, it can be done in the same way: 

ceph osd reweight $osdID $Weight

Where $osdID is the OSD number "52" and $Weight is a value up to 1:

ceph osd reweight 52 .90
    or
ceph osd reweight 52 .95
    or
ceph osd reweight 52 1

Monitor with:

ceph -s
ceph health detail
ceph osd df tree.  

Should the OSDs start to fill up again, re-weight the OSDs back down again. 

Note: osd.52 is now 81% full and has a re-weight value of "0.84999"

#==[ Command ]======================================#
# /usr/bin/ceph --connect-timeout=5 osd df tree
ID CLASS WEIGHT    REWEIGHT SIZE    USE     AVAIL   %USE  VAR  PGS TYPE NAME                    
-1       335.08347        -  335TiB  168TiB  167TiB 50.25 1.00   - root default 
52   ssd   0.72769  0.84999  745GiB  606GiB  140GiB 81.27 1.41  18         osd.52               
57   ssd   0.727#==[ Command ]======================================#
# /usr/bin/ceph --connect-timeout=5 osd df tree
ID CLASS WEIGHT    REWEIGHT SIZE    USE     AVAIL   %USE  VAR  PGS TYPE NAME                    
-1       335.08347        -  335TiB  168TiB  167TiB 50.25 1.00   - root default 
52   ssd   0.72769  0.84999  745GiB  606GiB  140GiB 81.27 1.41  18         osd.52               
57   ssd   0.72769  1.00000  745GiB  436GiB  310GiB 58.46 1.01  19         osd.57  
69  1.00000  745GiB  436GiB  310GiB 58.46 1.01  19         osd.57  

When the administrator is confident issue is resolved, remove the "noout" flag:   

ceph osd unset noou