cook: expand dither_tab[], and make sure indexes into it don't overflow.

Message ID 20120310200825.352AF5E033@aruru.libav.org
State New
Headers show

Commit Message

Janne Grunau March 10, 2012, 8:08 p.m.
Module: libav
Branch: master
Commit: 442c3a8cb1785d74f8e2d7ab35b1862b7088436b

Author:    Ronald S. Bultje <rsbultje@gmail.com>
Committer: Ronald S. Bultje <rsbultje@gmail.com>
Date:      Thu Mar  8 17:09:27 2012 -0800

cook: expand dither_tab[], and make sure indexes into it don't overflow.

Fixes overflows in accessing dither_tab[].

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org

---

 libavcodec/cook.c     |    6 +++++-
 libavcodec/cookdata.h |    4 ++--
 2 files changed, 7 insertions(+), 3 deletions(-)

Patch

diff --git a/libavcodec/cook.c b/libavcodec/cook.c
index 65e16e4..41ce0e5 100644
--- a/libavcodec/cook.c
+++ b/libavcodec/cook.c
@@ -507,7 +507,11 @@  static inline void expand_category(COOKContext *q, int *category,
 {
     int i;
     for (i = 0; i < q->num_vectors; i++)
-        ++category[category_index[i]];
+    {
+        int idx = category_index[i];
+        if (++category[idx] >= FF_ARRAY_ELEMS(dither_tab))
+            --category[idx];
+    }
 }
 
 /**
diff --git a/libavcodec/cookdata.h b/libavcodec/cookdata.h
index 126010a..c4c26fa 100644
--- a/libavcodec/cookdata.h
+++ b/libavcodec/cookdata.h
@@ -36,8 +36,8 @@  static const int expbits_tab[8] = {
     52,47,43,37,29,22,16,0,
 };
 
-static const float dither_tab[8] = {
-  0.0, 0.0, 0.0, 0.0, 0.0, 0.176777, 0.25, 0.707107,
+static const float dither_tab[9] = {
+  0.0, 0.0, 0.0, 0.0, 0.0, 0.176777, 0.25, 0.707107, 1.0
 };
 
 static const float quant_centroid_tab[7][14] = {